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EXECUTIVE  SUMMARY 


The  purpose  of  this  report  is  to  furnish  suggestions  for  improvement  of  the  Air  Mobility  Command 
(AMC)  Crew  Resource  Management  (CRM)  Anonymous  Reporting  System  (ARS)  reporting  form  based  on 
current  scientific  literature.  This  report  describes  CRM  dimensions  and  CRM  evaluation  techniques  helpful  in 
refining  the  AMC  CRM  ARS  form.  The  AMC  CRM  ARS  form  is  compared  to  research  findings,  and 
recommendations  for  revisions  to  the  ARS  form  are  provided.  In  summary,  this  report: 

•  reviews  current  literature  regarding  CRM 

•  identifies  concurrent  CRM  dimensions  common  to  the  Federal  Aviation  Administration  (FAA),  the  US  Air 
Force  (USAF),  and  mainstream  CRM  literature 

•  identifies  applicable  CRM  rating  methods 

•  evaluates  the  current  AMC  CRM  ARS  reporting  form  using  widely  accepted  CRM  dimensions  and  rating 
methods 

•  presents  recommendations  for  the  improvement  of  the  AMC  CRM  ARS  reporting  form 

To  quantify  CRM,  the  Crew  System  Ergonomics  Information  Analysis  Center  (CSERIAC)  first 
defined  the  concept,  and  then  decomposed  CRM  into  individual  dimensions.  Since  this  report  used  current 
mainstream  research  literature,  it  was  necessary  to  ensure  that  the  mainstream  and  USAF  CRM  definitions  and 
dimensions  concurred  with  each  other.  CSERIAC  compared  and  contrasted  a  variety  of  mainstream  CRM 
dimensions  with  the  USAF  dimensions.  Dimensions  that  appeared  across  taxonomies  were  identified  as 
defensible  CRM  dimensions,  while  dimensions  that  had  no  concurrence  with  other  taxonomies  were  identified 
for  revision. 

CSERIAC  reviewed  the  Line/LOS  Checklist  (LLC),  the  Aircrew  Coordination  Evaluation  (ACE) 
Checklist,  and  the  AMC  CRM  Assessment  Sheet  to  determine  appropriate  means  of  assessing  CRM  behaviors. 
From  this  review,  it  appears  appropriate  to  use  CRM  behavioral  marker  ratings  which  are  mapped  back  to 
distinct,  well-defined  CRM  curriculum  elements.  A  broad  taxonomy  of  possible  behavioral  markers  is 
provided  in  the  present  report.  Ideally,  the  same  behavioral  markers  that  are  used  in  aircrew  CRM  training 
should  also  be  used  in  operational  reporting. 

The  12  factors  appearing  on  the  current  AMC  CRM  ARS  form  address  CRM-specific,  general  crew 
experience,  and  general  event  context  information  in  the  same  section.  CSERIAC  recommends  that  the  factors 
which  actually  address  CRM  on  the  ARS  form  be  separated  from  those  that  do  not  directly  address  CRM 
dimensions  of  an  event. 

In  conclusion,  CSERIAC  recommends  the  revision  and  reorganization  of  the  current  AMC  CRM  ARS 
reporting  form.  In  its  present  state,  the  form  does  not  appear  to  map  well  to  Air  Force  Instruction  (AFI)  36- 
2243  (including  AMC  Supplement  1),  accepted  taxonomies  of  CRM,  or  CRM  behavioral  markers. 
Additionally,  the  form  presents  individual  factors  that  overlap  heavily  across  several  distinct  CRM  dimensions. 
CSERIAC  recommends  that  the  ARS  form  be  extensively  revised  and  divided  into  the  following  five  sections: 

•  Contact  information 

•  Crew  background  information 

•  Event  context  information 

•  CRM  behavioral  markers 

•  Narrative  description 

Contact  information  would  simply  include  name,  address,  and  telephone  numbers.  The  crew 
experience  section  would  include  information  on  the  crew’s  training,  proficiency,  and  background.  The  event 
context  section  would  provide  weather,  conflict,  and  equipment  failure  information  (along  with  other  context- 
related  data)  of  the  situation  being  reported.  The  CRM  behavioral  marker  section  would  address  the  CRM- 
specific  elements  surrounding  the  situation  being  reported  by  allowing  the  reporter  to  rate  (on  Likert  scale) 
specific,  observable  behaviors  which  can  be  readily  traced  back  to  USAF  CRM  dimensions.  Finally,  the 
narrative  description  section  would  allow  AMC  CRM  ARS  reporters  to  freely  write  an  account  of  the  reported 
situation  from  start  to  finish,  providing  suggested  improvements  where  appropriate. 
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1.  INTRODUCTION 


1.1  Background 

In  1972,  a  Lockheed  L-101 1  descended  at  night  into  the  trees  and  swamps  of  the 
Florida  Everglades,  killing  99  passengers  and  crewmembers  on  board  (NTSB,  1973).  During 
the  ensuing  National  Transportation  Safety  Board  (NTSB)  investigation,  it  was  revealed  that 
one  small,  bumed-out  landing  gear  indicator  set  in  motion  a  sequence  of  events  that  ended  in 
disaster.  More  accurately,  the  response  of  the  cockpit  crew  to  the  inoperative  bulb  ultimately 
hardened  the  last  links  in  this  chain  of  catastrophe.  While  in  flight,  each  crewmember  (flight 
engineer,  first  officer,  and  captain)  fixated  on  solving  the  same  condition,  an  aberrant  landing 
gear  down  and  locked  bulb,  neglecting  to  notice  that  the  autopilot  had  become  disengaged. 
Quietly,  while  all  crewmembers  were  attending  to  the  same,  non-emergency  condition,  the 
aircraft  descended,  under  neither  human  nor  automatic  control,  until  it  finally  augured  in  to 
the  mud  and  water  below. 

Twenty-four  years  later,  we  identify  the  Everglades  L-101 1  accident  as  a  classic  study 
in  Crew  Resource  Management  (CRM)  deficiencies.  Neither  the  United  States  (US)  airlines 
nor  the  United  States  Air  Force  (USAF)  were  formally  trained  in  CRM  in  the  1970’s  (indeed, 
the  concept  was  not  specifically  developed  until  the  late  1970’s).  CRM  was  first  (arguably) 
implemented  by  United  Air  Lines  in  1979  as  noted  in  Keyes  (1990).  It  is  important  to 
realize,  however,  that  for  some  airlines,  CRM  is  simply  a  new  name  for  a  crew  concept  that 
has  always  been  trained.  Consider  this  excerpt  from  Massey  (1990): 

Not  every  airline  feels  that  CRM  is  really  new.  TWA  [Trans  World  Airlines], 
which  calls  its  current  program  Cockpit  Resource  Training,  has  long  stressed 
the  crew  concept,  according  to  a  training  department  representative.  “We’ve 
been  doing  CRM  for  years  and  years,  although  we  haven’t  been  calling  it 
that,”  he  said.  “I’ve  been  with  the  company  for  25  years,  and  I  think  the  crew 
concept’s  been  around  longer  than  that.”  (p.  65) 

Today,  CRM  is  a  widely  accepted  training  concept  and  program  in  major  airlines  and  the 
USAF.  Current  CRM  initiatives  involve  the  refinement  and  improvement  of  CRM  training 
programs  with  the  ultimate  goals  of  improving  safety,  enhancing  mission  effectiveness,  and 
augmenting  training  efficiency  (USAF,  1994).  These  initiatives  seek  to  operationally  fulfill 
the  USAF  (1994)  CRM  definition:  “the  effective  use  of  all  available  resources — ^people, 
weapon  systems,  facilities,  and  equipment,  and  environment — ^by  individuals  or  crews  to 
safely  and  efficiently  accomplish  an  assigned  mission  or  task”  (p.  14). 

1.2  Purpose  of  this  Report 

This  report  provides  suggestions  for  possible  changes  to  the  Air  Mobility  Command 
(AMC)  CRM  Anonymous  Reporting  System  (ARS)  reporting  form.  The  AMC  CRM  ARS  is 
a  program  initiated  by  AMC  to  augment  the  CRM  training  process.  The  system  consists  of  a 
questionnaire  on  which  aircrew  members  supply  information  about  a  CRM  incident  and  a 
database  in  which  questionnaire  data  is  stored.  Headquarters  Air  Mobility  Command, 
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Directorate  of  Operations  and  Training  (HQ  AMC/DOT)  continuously  requests  that  its 
aircrew  members  complete  the  questionnaire  when  they  experience  or  observe  a  CRM  event, 
be  it  positive  or  negative.  AMC  indicates  that  the  information  gained  through  this 
anonymous  source  will  be  used  to  improve  operations  and  training  programs.  It  is  essential 
for  aircrew  members  to  report  CRM  events,  and  it  is  just  as  important  to  have  a  means  of 
classifying  incidents.  Appropriate  classification  of  CRM  incidents  can  serve  as  a  pointer  to 
areas  that  might  benefit  from  increased  training  emphasis  or  implementation  of  new  training 
methods. 

The  purpose  of  this  report  is  to  evaluate  the  current  AMC  CRM  ARS  reporting  form 
in  light  of  the  current  literature  and  research  on  CRM.  The  objectives  are: 

•  Review  current  literature  regarding  CRM. 

•  Identify  concurrent  CRM  dimensions  common  to  the  Federal  Aviation  Administration 
(FAA),  the  US  Air  Force  (USAF),  and  mainstream  CRM  literature.. 

•  Identify  applicable  CRM  rating  methods. 

•  Evaluate  the  current  AMC  CRM  ARS  reporting  form  using  widely  accepted  CRM 
dimensions  and  rating  methods. 

•  Present  recommendations  for  the  improvement  of  the  AMC  CRM  ARS  reporting  form. 

1.3  Scope  of  this  Report 

Although  the  design  recommendations  presented  within  this  report  provide  a  basis 
upon  which  to  improve  the  AMC  CRM  ARS  reporting  form,  the  present  effort  does  not 
provide  a  finished  product  form. 
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2.  METHOD 


In  order  to  provide  defensible  suggestions  for  the  improvement  of  the  AMC  CRM 
ARS  reporting  form,  the  Crew  System  Ergonomics  Information  Analysis  Center  (CSERIAC) 
elected  to  systematically  analyze  elements  believed  to  be  important  to  an  effective  ARS  form. 
Broadly,  this  analysis  consisted  of  three  parts,  including  an  analysis  of  overall  CRM 
dimensions,  an  analysis  of  observable  behaviors  constituting  these  dimensions,  and  an  overall 
analysis  of  the  current  AMC  CRM  ARS  form  and  structure.  These  three  analysis 
components  are  described  below,  and  each  analysis  component  composes  a  major  section 
(i.e.,  3.1,  3.2,  and  3.3)  in  the  body  of  this  report. 

2.1  Analysis  of  CRM  Dimensions 

In  order  to  quantify  CRM,  it  was  necessary  to  first  define  the  concept,  and  then  break 
CRM  down  into  individual  dimensions.  Since  this  report  used  current  mainstream  research 
literature,  it  was  necessary  to  compare  and  determine  the  extent  to  which  mainstream  (e.g., 
FAA,  1995)  and  USAF  CRM  (USAF,  1994;  USAF,  1995)  definitions  and  dimensions 
concurred  with  each  other. 

CSERIAC  compared  and  contrasted  a  variety  of  mainstream  CRM  dimensions  with 
the  Air  Force  dimensions.  USAF  dimensions  that  appeared  across  mainstream  CRM 
taxonomies  were  identified  as  defensible  CRM  dimensions,  while  USAF  CRM  dimensions 
that  had  no  concurrence  with  mainstream  dimensions  were  identified  for  revision.  CSERIAC 
also  considered  the  possibility  that  some  mainstream  dimensions  might  not  appear  among 
USAF  dimensions.  However,  this  was  not  the  case. 

2.2  Analysis  of  Crew  Performance  Evaluation  Techniques 

Although  breaking  CRM  down  into  defensible  dimensions  was  an  important  first  step 
in  developing  an  effective  assessment  of  CRM,  CSERIAC  realized  that  crewmembers  could 
find  difficulty  in  distinguishing  among  CRM  dimensions  unless  the  dimensions  were  broken 
down  into  behavioral  definitions.  A  behavioral  definition  for  a  dimension  consists  of  a  list  of 
specific,  observable  behaviors  that  represent  that  dimension 

CSERIAC  identified  two  lists  of  behavioral  markers  (fi'om  FAA  and  the  AMC)  and 
compared  these  lists.  A  new  list  consisting  of  the  sum  of  these  lists  was  then  formed. 
CSERIAC  compared  this  new  list  of  behavioral  markers  with  the  current  ARS  form,  pointing 
out  possible  improvements  to  the  form.  CSERIAC  also  reorganized  the  behavioral  markers 
to  map  back  to  the  specific,  defensible  USAF  CRM  dimensions  identified  in  the  analysis  of 
CRM  dimensions  (2.1,  above). 

2.3  Analysis  of  Present  AMC  CRM  ARS  Form  and  Format 

Finally,  CSERIAC  reviewed  the  overall  content  and  structure  of  the  AMC  CRM  ARS 
form.  CSERIAC  compared  the  ARS  form  to  the  National  Aeronautics  and  Space 
Administration  (NASA)  Aviation  Safety  Reporting  System  (ASRS)  form  and  commented  on 
possible  structure  and  content  changes  resulting  from  this  comparison.  Specific  to  CRM 
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assessment  of  the  ARS  form,  CSERIAC  compared  the  12  factors  presently  appearing  on  the 
AMC  CRM  ARS  form  to  the  CRM  dimensions  (2.1)  and  crew  performance  markers  (2.2) 
previously  identified. 
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3.  RESULTS  AND  DISCUSSION 


The  purpose  of  this  section  is  to  review  current  research  findings  as  they  pertain  to  the 
present  evaluation  of  the  AMC  CRM  ARS  reporting  form.  These  research  findings  help 
emphasize  and  organize  dimensions  for  describing  CRM  functions. 

3.1  Analysis  of  CRM  Dimensions 

3.1.1  Purpose  and  Definition  of  CRM 

The  primary  purpose  of  CRM  is  to  enhance  flight  safety.  A  large  number  of  accidents 
due  to  lack  of  crew  coordination  (e.g.,  Everglades  L-101 1  accident)  raised  concerns  with 
several  organizations,  including  the  airlines,  the  FAA,  the  NTSB,  and  the  USAF.  Most  CRM 
programs  exist  for  the  principal  reason  of  enhancing  safety.  Air  Force  Instruction  (AFI)  36- 
2243  (USAF,  1994)  specifies  a  total  of  three  objectives  for  the  USAF  CRM  program.  These 
objectives  include  maximizing  operational  effectiveness  and  combat  capability,  preventing 
incidents  and  accidents  to  preserve  human  and  materiel  resources,  and  improving  efficiency 
of  all  training.  Although  safety  is  at  the  crux  of  the  CRM  program,  it  follows  that  increases 
in  crew  coordination  will  not  only  increase  safety,  but  also  enhance  operational  effectiveness 
and  combat  capability. 

To  enhance  the  utility  of  the  CRM  ARS  improvement  effort,  it  is  important  that  ARS 
reporters  and  analysts  share  the  same  perceptions  and  definitions  regarding  CRM.  Consider 
the  opinion  of  Driskell  and  Adams  (1992),  which  summarizes  the  generally  accepted  main 
themes  of  CRM: 

Crew  resource  management  represents  an  approach  to  improving  aviation 
safety  that  was  bom  of  real  life  experiences  of  airline  pilots.  They  realized 
that  technical  skill  alone  was  not  enough  to  manage  safely  a  complex  flight 
system.  CRM  emphasizes  the  effective  utilization  of  all  resources  available  to 
the  flight  crew,  including  equipment  and  people.  In  addition  to  respecting  the 
importance  of  traditional  stick  and  mdder  skills,  CRM  focuses  on  those  other 
skills  required  for  effective  crew  performance.  The  overall  goal  of  CRM  is 
the  blending  of  technical  skills  and  human  skills  so  as  to  support  safe  and 
efficient  operation  of  aircraft,  (p.  10,  original  emphasis) 

CRM  is  broadly  conceptualized  as  the  proper  utilization  of  all  available  resources— 
hardware,  software,  and  liveware-to  achieve  safe,  efficient  flight  operations  (Tauber,  1987). 
For  Tauber  (1987),  hardware  includes  the  aircraft  itself,  the  instmments,  and  ground 
equipment.  Software  (in  contrast  to  its  use  in  the  computer  domain)  refers  to  procedures, 
planning,  and  general  policies.  Tiveware  refers  to  the  people  available  in  the  system. 

Tauber’s  definition  has  been  divided  further  into  goals  and  objectives  by  researchers  and 
organizations,  including  the  FAA  (FAA,  1995). 
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3.1.2  FAA  Advisory  Circular  120-51B  Suggested  CRM  Dimensions 

The  FAA  Advisory  Circular  (AC)  120-5 IB  (FAA,  1995)  is  a  landmark  publication  on 
CRM  training,  and  suggests  basic  CRM  skills  appropriate  for  use  in  developing  CRM 
training  programs  (Driskell  and  Adams,  1992).  The  FAA  categorizes  CRM  into  three  main 
clusters,  including  (1)  communication  processes  and  decision  behavior,  (2)  team  building  and 
maintenance,  and  (3)  workload  management  and  situation  awareness.  These  clusters  are 
further  broken  down  into  subtopics  which  can  be  more  readily  trained  and  assessed  than  the 
overall  clusters  (see  Table  1). 


Table  1 

FAA  AC  120-51B  Suggested  CRM  Topics  (FAA,  1995,  pp.  10-A6) 


Cluster 

Subtopics 

Communication 
Processes  and 
Decision  Behavior 

1 .  Briefings 

2.  Inquiry/Advocacy/Assertion 

3.  Crew  Self-Critique 

4.  Conflict  Resolution 

5.  Communications  and  Decision  making 

Team  Building  and 
Maintenance 

1 .  Leadership/Followership/Concern  for  Tasks 

2.  Interpersonal  Relationships/Group  Climate 

Workload 

Management  and 
Situation  Awareness 

1 .  Preparation/PlanningA/igilance 

2.  Workload  Distributed/Distractions  Avoided 

3.  Individual  Factors/Stress  Reduction 

From  this  division  into  clusters  and  subtopics,  FAA  (1995)  develops  a  list  of 
behavioral  crew  performance  markers.  Essentially,  this  is  a  list  of  observable  behaviors  that 
may  assist  in  developing  CRM  programs  and  aid  in  providing  guidelines  for  evaluation  and 
feedback.  Each  behavioral  marker  can  be  mapped  back  to  a  specific  subtopic  classification 
(e.g.,  preparation/planning/vigilance).  Although  not  presented  as  a  checklist  for  evaluating 
individual  crevwnembers,  these  behavioral  performance  markers  could  form  the  basis  for  an 
evaluation  strategy  adapted  to  the  AMC  CRM  ARS  form  refinement  purpose.  The  FAA’s 
crew  performance  marker  clusters  are  provided  in  Appendix  A  of  this  report. 

3.1.3  Concurrent  Lists  of  CRM  Dimensions 

Many  researchers  and  organizations  share  common  beliefs  on  the  definition,  purpose, 
and  desired  outcomes  of  CRM.  Consider  the  CRM  dimensions  used  by  various  authors  and 
organizations  (Table  2)  and  note  the  similarities  among  them.  FAA  AC  120-5 IB  dimensions 
have  been  integrated  into  this  list  for  easier  comparison  of  common  themes.  Recurrent 
themes  appear  among  the  various  dimensions,  lending  concurrent  validity  to  the  factors 
purported  to  compose  the  CRM  construct. 
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Table! 

CRM  Dimensions  Used  by  Other  Authors  and  Organizations 


Dimensions 

Reference 

1 .  Briefings 

FAA  AC  120- 

2 .  I  nq  u  iry/Advocacy/Assertion 

51 B, 1995 

3.  Crew  Self-Critique 

(Regulatory 

4.  Conflict  Resolution 

Organization) 

5.  Communications  and  Decisionmaking 

6.  Leadership/Followership/Concern  for  Tasks 

7.  Interpersonal  Relationships/Group  Climate 

8.  Preparation/PlanningA/igllance 

9.  Workload  Distributed/Distractions  Avoided 

10.  Individual  Factors/Stress  Reduction 

1 .  To  ensure  that  established  rules  and  procedures  are  followed  routinely  and  without  exception 

CRM  Training 

2.  To  make  the  “crew  concept”  in  practice  do  justice  to  the  concept 

Goals  from 

3.  To  maintain  a  high  level  of  flight  safety  awareness  in  ail  flightcrew  members 

Orlady,  1983 

4.  To  take  constructive  advantage  of  operational  incidents 

(Airline) 

1 .  Appropriate  delegation  of  tasks  and  assignments 

American 

2.  Establishment  of  a  logical  order  of  priorities 

Airlines’  Seven 

3.  Continuous  monitoring  and  cross  checking  of  essential  Instruments  and  systems 

Principles  of 

4.  Careful  assessment  of  problems  and  avoidance  of  preoccupation  with  minor  ones 

Flight  Deck 

5.  Utilization  of  all  available  data  to  conduct  an  operation 

Resource 

6.  Clear  communication  of  all  plans  and  intentions  among  crewmembers 

Management; 

7.  Assurance  of  sound  leadership  by  the  pilot  in  command 

Telfer,  1983 

(Airline) 

1 .  Delegating  tasks  and  assigning  responsibilities 

Fundamental 

2.  Establishing  priorities 

Elements  of 

3.  Monitoring  and  cross-checking 

Effective  CRM 

4.  Assessing  problems  and  avoiding  preoccupation 

from  Steenblik, 

5.  Communicating 

1988 

6.  Leadership 

(Airline) 

1 .  Developing  effective  Interpersonal  communication  styles 

Basic  CRM 

2.  Developing  leadership/followership 

Concepts  from 

3.  Developing  decision  making  skills 

Jensen,  1989 

4.  Developing  a  “team”  concept 

(Airline) 

5.  Dealing  with  stress 

1 .  Attention  management 

Cockpit 

2-  Crew  management 

Management 

3.  Stress  management 

Tools  from 

4.  Attitude  management 

Diehl,  1991 

5.  Risk  management 

(Military) 

1.  Decisionmaking 

Team  Process 

2.  Assertiveness 

Skills  from 

3.  Mission  analysis 

Prince, 

4.  Communication 

Brannick, 

5.  Leadership 

Prince,  &  Salas, 

6.  Adaptability/flexibility 

1992 

7.  Situational  awareness 

(Military) 
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3.1.4  Comparison  of  Mainstream  and  Air  Force  CRM  Dimensions 

In  order  to  apply  the  lessons  learned  found  in  mainstream  CRM  research  to  the 
special  operating  environment  of  the  AMC,  it  is  necessary  to  confirm  that  the  generally 
accepted  dimensions  of  CRM  concur  with  the  CRM  dimensions  defined  by  the  USAF  and 
the  AMC.  AFI  36-2243  (USAF,  1994)  is  the  official  doctrine  detailing  the  CRM  program 
objectives  in  the  USAF,  and  AMC  Supplement  1  (USAF,  1995)  to  AFI  36-2243  adds  AMC- 
specific  doctrine  to  the  original  instruction.  The  comparative  analysis  of  mainstream  and 
USAF  CRM  dimensions  is  provided  in  Table  3.  In  column  1,  the  USAF  CRM  dimension  is 
presented.  Its  associated  definition  appears  in  column  2,  and  column  3  provides  the 
mainstream  comparable  dimensions. 

The  core  curriculum  mandated  in  AFI  36-2243  requires  the  implementation  of  CRM 
training  in  eight  different  areas.  These  eight  AFI  3 6-2243 -specific  dimensions  are  identified 
in  Table  3  by  normal  (not  bold)  print.  Notice  that  these  dimensions  generally  agree  with  the 
CRM  dimensions  originally  shown  in  Table  2. 

In  Supplement  1  to  AFI  36-2243,  AMC  has  delineated  an  additional  six  required 
CRM  subjects  (also  found  in  Table  3)  as  AMC-specific  core  curriculum  matters.  These 
Supplement  1 -specific  subjects  are  identified  by  bold  print.  Note  that  these  six  core 
curriculum  amplify  the  elements  widely  accepted  as  CRM  issues  (e.g.,  assertiveness  and 
effective  communications).  For  the  present  report,  the  AMC  CRM  dimensions  will  be 
considered  as  additions  to  AFI  36-2243,  not  replacements  for  this  instruction.  The  AMC 
CRM  dimensions  act  to  augment  the  original  instruction  by  more  succinctly  delineating  and 
defining  AFI  36-2243  dimensions. 

Most  of  the  USAF  CRM  dimensions  concur  with  the  mainstream  CRM  dimensions. 
Two  of  the  USAF  and  AMC  CRM  dimensions,  however,  are  not  supported  by  the  literature 
reviewed  for  this  report.  Namely,  human  performance  (AFI  36-2243)  and  concepts  of 
mishap  prevention  (Supplement  1)  were  found  nowhere  else  in  the  literature  as  dimensions  of 
CRM.  The  primary  reason  for  this  exclusion  is  that  both  dimensions  are  too  broad  for 
inclusion  as  subclassifications  of  CRM.  For  instance,  the  human  performance  dimension 
overtly  overlaps  two  already  existing  concepts:  situational  awareness  and  stress  awareness 
and  management  (and  when  further  analyzed,  human  performance  also  overlaps  several  other 
sufficiently  distinct  CRM  dimensions).  The  concepts  of  mishap  prevention  dimension  also 
overlaps  situational  awareness  and  is  once  again  combined  with  several  other  topic  areas. 
Additionally,  it  is  easily  argued  that  CRM  itself  is  the  operational  application  of  concepts  of 
mishap  prevention,  thus  illustrating  the  unacceptably  wide  scope  that  concepts  of  mishap 
prevention  takes  on  when  categorized  as  a  dimension  within  CRM. 

This  is  not  to  say  that  these  subject  areas  are  not  worthwhile.  However,  their  inherent 
broadness  will  induce  difficulty  in  CRM  assessment.  For  instance,  if  a  returned  CRM  ARS 
form  contained  responses  suggesting  a  problem  with  CRM  human  performance  training,  it 
would  be  challenging  to  ascertain  exactly  where  the  problem  lies-is  the  training  deficiency 
in  stress  awareness  and  management  training,  or  in  situational  awareness  training? 
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Tables 

Comparison  of  USAF  and  Mainstream  CRM  Dimensions 


Concept 

Description 

Supporting  References 

Situational 

Awareness 

A  desired  end  state  of  CRM  training  is  a  high 
state  of  situational  awareness.  Tools  for 
preventing  lost  situational  awareness,  cues 
for  recognizing  lost  situational  awareness, 
techniques  for  recovering  from  lost  situational 
awareness  will  be  covered  under  this 
concept  area. 

Situation  awareness  subtopic  (FAA,  1995) 

Attention  management  (Diehl,  1991) 

Situational  awareness  (Prince  et  aL,  1992) 

Avoidance  of  preoccupation  (Telfer,  1983) 

Utilization  of  all  available  data  (Telfer,  1983) 
Avoidance  of  preoccupation  (Steenblik,  1988) 

Group  Dynamics 

Includes  command  authority,  leadership, 
responsibility,  assertiveness,  conflict 
resolution,  hazardous  attitudes,  behavioral 
styles,  legitimate  avenues  of  dissent,  team¬ 
building,  and  desired  traits. 

Leadership  subtopic  (FAA,  1995) 

“Crew  concept”  do  justice  (Orlady,  1983) 

Crew  management  (Diehl,  1991) 

Developing  “team”  concept  (Jensen,  1989) 

Sound  leadership,  delegation  (Telfer,  1983) 
Leadership,  delegating  (Steenblik,  1988) 

Developing  leadership/followership  (Jensen,  1989) 
Assertiveness,  leadership  (Prince  et  al.,  1992) 

Group 

Dynamics 

Training 

Behavior  norms  of  groups  and 
individuals,  team  building,  leadership 
styles,  synergy  demonstrations,  and 
interactive  exercises  for  all  crewmembers. 

Leadership  subtopic  (FAA,  1995) 

“Crew  concept”  do  justice  (Orlady,  1983) 

Crew  management  (Diehl,  1991) 

Developing  “team”  concept  (Jensen,  1989) 

Sound  leadership,  delegation  (Telfer,  1983) 
Leadership,  delegating  (Steenblik,  1988) 

Develop  leadership/followership  (Jensen,  1989) 
Assertiveness,  leadership  (Prince  et  al.,  1992) 

Effective 

Communications 

Includes  common  errors,  cultural  influences, 
and  barriers  such  as  rank,  age,  and  position, 
participation  of  all  crewmembers.  Also  stress 
coordination  with  other  participants  in  a 
mission,  interface  concerns,  listening, 
feedback,  precision  and  efficiency  of 
communication. 

Communication  subtopic  (FAA,  1995) 

Clear  communication  (Telfer,  1983) 

Communicating  (Steenblik,  1988) 

Interpersonal  communication  styles  (Jensen,  1989) 
Communication  (Prince  et  al.,  1992) 

Effective 

Communications 

Importance  of  effective  communication, 
barriers  to  communication,  active 
listening,  nonverbal  communication,  and 
challenge  and  response. 

Communication  subtopic  (FAA,  1995) 

Clear  communication  (Telfer,  1983) 
Communicating  (Steenblik,  1988) 

Interpersonal  communication  styles  (Jensen, 

1989) 

Communication  (Prince  et  al.,  1992) 

Assertiveness 

Training 

impact  of  personality  differences, 
empower  to  challenge,  authority  with 
participation,  assertiveness  with  respect, 
inquiry,  advocacy,  and  assertion. 

Assertiveness  (Prince  et  al.,  1992) 
Inquiry/advocacy/assertion  (FAA,  1995) 

Risk 

Management  and 
Decision  Making 

This  risk  assessment  and  risk  management 
styles,  process,  tools,  breakdowns  in 
judgment  and  discipline,  problem-solving, 
evaluation  of  hazards,  management  of 
regulatory  deviation  during  emergencies,  and 
the  most  conservative  response  rule. 

Decision  making  subtopic  (FAA,  1995) 

Flight  Safety  awareness  (Orlady,  1983) 

Decision  making  (Prince  et  al.,  1992) 

Decision  making  skills  (Jensen,  1989) 

Risk  management  (Diehl,  1991) 

Assessing  problems  (Telfer,  1983) 

Decision 

Making 

Recognize  need  for  action,  gather 
information,  evaluate  course  of  action, 
select  and  implement  the  decision, 
evaluate  the  decision,  risk  management. 

Decision  making  subtopic  (FAA,  1995) 

Decision  making  (Prince  et  al.,  1992) 

Decision  making  skills  (Jensen,  1989) 

Assessing  problems  (Telfer,  1983) 

Flight  safety  awareness  (Orlady,  1983) 

Workload 

Management 

This  area  covers  overload,  underload, 
complacency,  management  of  automation, 
available  resources,  checklist  discipline,  and 
standard  operating  procedures. 

Workload  management  subtopic  (FAA,  1995) 

Follow  rules  and  SOPs  (Orlady,  1983) 

Monitoring  and  cross-checking  (Telfer,  1983) 
Monitoring  and  cross-checking  (Steenblik,  1988) 
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Tables  (Continued) 

Comparison  of  USAF  and  Mainstream  CRM  Dimensions 


Concept 

Description 

Supporting  References 

Stress 

Awareness  and 
Management 

Includes  sources  of  stress,  benefits  and 
hazardous  effects,  and  coping  techniques. 

Stress  subtopic  (FAA,  1995) 

Dealing  with  stress  (Jensen,  1989) 

Stress  management  (Diehl,  1991) 
Adaptability/flexibility  (Prince  et  al.,  1992) 

Stress 

Management 

Recognizing  the  signs  of  stress,  signs  of 
stress  on  the  individual,  signs  of  stress  in 
others,  positive  and  negative  stress, 
common  methods  for  stress  reduction, 
and  workload  distribution. 

Stress  subtopic  (FAA,  1995) 

Dealing  with  stress  (Jensen,  1989) 

Stress  management  (Diehl,  1991) 

Workload  management  subtopic  (FAA,  1995) 

Mission, 

Planning, 

Review,  and 
Critique 

Strategies 

This  area  covers  pre  mission  analysis  and 
planning,  bnefing,  ongoing  or  mid-mission 
review,  and  post  mission  critique.  This  area 
is  intended  to  allow  individual  MAJCOMs  and 
FOAs  to  include  mission  specific  and  weapon 
system  specific  planning,  briefing  and  critique 
tools  in  their  CRM  training  programs. 

Critique  subtopic  (FAA,  1995) 

Constructive  advantage  of  incidents  (Orlady,  1983) 
Mission  analysis  (Prince  et  al.,  1992) 

Logical  order  of  priorities  (Telfer,  1983) 

Establishing  priorities  (Steenblik,  1988) 

Human 

Performance 

This  includes  situational  awareness, 
cognitive  processing,  anomalies  of  attention, 
stress  and  stress  management,  behavioral 
styles,  mental  attitudes,  fatigue  effects. 

These  aviation  specific  knowledge  and  skills 
are  taught  by  operationally  experienced 
aviation  psychologists  and  physiologists  and 
may  be  taught  in  conjunction  with 
physiological  training. 

Attitude  management  (Diehl,  1991) 

Concepts  of 

Mishap 

Prevention 

The  error  chain,  attitude  management, 
hazardous  attitudes,  hazardous 
behaviors,  human  nature  pitfalls, 
situational  awareness,  elements  of  good 
situational  awareness,  and  symptoms  and 
causes  of  decreased  situational 

awareness. 

Attitude  management  (Diehl,  1991) 

Situation  awareness  subtopic  (FAA,  1995) 
Situational  awareness  (Prince  et  al.,  1992) 

3.2  Crew  Performance  Evaluation  Techniques 
3.2.1  Attitude  Questionnaires 

While  specific,  well-defined  dimensions  are  essential  to  the  assessment  of  CRM,  each 
dimension  must  be  further  analyzed  and  items  that  may  be  rated  developed.  Several 
strategies  have  been  proposed  regarding  the  nature  of  these  ratable  items.  Murray,  Weeks, 
and  Siem  (1995)  provide  a  summary  of  the  current  state  of  CRM  evaluation  methods: 

To  date,  CRM  assessment  in  civilian  aviation  has  consisted  largely  of 
measurement  of  CRM  attitudes  with  instruments  such  as  the  Cockpit 
Management  Attitudes  Questionnaire  (CMAQ;  Helmreich,  Wilhelm  & 

Gregorich,  1991).  Similarly,  in  the  Air  Force,  CRM  assessment  focuses  on 
student  attitudes.  CRM  is  considered  a  “state  of  mind”  and  training 
emphasizes  the  importance  of  fostering  attitudes  that  support  CRM. 

Although  assessment  of  attitudes  provides  important  information  that 
helps  identify  training  needs  and  student  receptivity  to  training,  the 
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assumption  that  attitudes  directly  influence  actual  behaviors  is  questionable 
(Fishbein  &  Azjen,  1975).  As  a  result,  requirements  for  measuring  CRM 
skills  and  behaviors  have  emerged.  Programs  such  as  the  Federal  Aviation 
Administration  Advanced  Qualification  Program  (AQP;  FAA,  1991)  and  the 
Air  Force  Instruction  for  CRM  Training  (USAF,  1994)  reflect  the  increasing 
importance  of  CRM  evaluation  based  on  performance  rather  than  attitudes. 

(P-  1) 

At  the  conclusion  of  CRM  training,  students  are  often  given  an  attitude  questionnaire, 
which  is  used  to  assess  and  predict  the  effectiveness  of  the  CRM  training.  Although  useful 
for  the  immediate  purpose  of  gathering  subjective  student  opinion  (or  attitude)  on  CRM 
training,  the  real  measure  of  effective  CRM  training  is  operational  effectiveness  (a  behavioral 
criteria).  In  essence,  the  CRM  ARS  eoncept  is  perhaps  the  very  best  indicator  of  the  success 
of  CRM  programs  in  the  USAF.  A  crewmember  reporting  a  CRM  incident  momentarily 
takes  the  role  of  an  evaluator  or  rater.  The  evaluation  that  the  rater  provides  is  much  closer  to 
a  rating  of  people  and  performance  rather  than  attitude  about  the  CRM  aeeeptance. 

CSERJAC  recommends  that  the  CRM  Anonymous  Reporting  System  avoid  attitude 
measures  (not  described  in  the  present  report).  Attitude  measures  are  certcdnly  useful,  but 
considering  the  operational  incidents  from  which  the  CRM  ARS  responses  will  be  borne,  the 
most  usable  information  will  be  behavioral  information.  Behavioral  information  will  be 
more  apt  to  provide  clear,  visible  signs  of  both  excellent  and  poor  CRM.  Behavioral 
information  can  be  linked  to  the  skills  which  need  to  be  taught  and  emphasized  in  CRM 
courses.  Attitude  information,  on  the  other  hand,  may  not  directly  address  the  CRM 
dimensions  present  in  a  particular  CRM  event. 

Examining  the  major  checklists  used  during  training  to  rate  crewmembers’  outward, 
CRM-related  behavior  may  prove  useful  in  developing  a  checklist  for  rating  CRM  behaviors 
found  in  the  operational  environment.  The  two  major  behavioral  CRM  rating  checklists 
found  in  the  literature  are  described  below. 

3.2.2  Line/LOS  and  ACE  Checklists 

The  two  primary  behaviorally-anchored  expert  rating  forms  found  in  the  literature  are 
the  Line/LOS  (Line  Operational  Simulation)  Checklist  (LLC,  also  referred  to  as  the 
Line/LOFT  Checklist  in  some  reports),  and  the  Aircrew  Coordination  Evaluation  (ACE). 

The  LLC  was  developed  in  the  NASA/University  of  Texas  (UT)/FAA  Aerospace  Crew 
Research  Project  (Helmreich  and  Wilhelm,  1987),  while  the  ACE  was  developed  for  the 
Army  Research  Institute  Aviation  Research  and  Development  Activity  at  Fort  Rucker,  AL 
(Simon,  Risser,  Pawlik,  and  Leedom,  1990).  The  LLC  began  as  a  17-dimension  checklist 
utilizing  a  5-point  Likert  scale  (Povenmire,  Rockway,  Bunecke,  and  Patton,  1989;  Taggart 
and  Butler,  1989).  The  ACE  was  derived  from  this  work,  adding  2  dimensions  for  a  total  of 
1 9,  and  utilized  a  7-point  Likert  scale  rather  than  the  5-point  Likert  of  the  LLC.  See  Figure  1 
for  a  comparison  of  the  two  original  checklists.  Note  that  similar  behaviors  are  directly 
across  from  each  other,  except  where  an  arrow  points  to  the  equivalent  behavior(s). 
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Line/LOS  Checklist  (LLC) 


1. 


2. 


Briefing  thorough,  establishes  open  communications, 
addresses  coordination,  planning,  team  creation,  and 
anticipates  problems. 

Communications  timely,  relevant,  complete,  and  verified. 


3.  Inquiry/questioning  practiced. 

4.  Advocacy/assertion  practiced. 

5.  Decisions  communicated  and  acknowledged 

6.  Crew  self-critique  of  decisions  and  actions. 

7.  Concern  for  accomplishment  of  tasks  at  hand 

8.  Interpersonal  relationships/group  climate. 

9.  Overall  vigilance. 

10.  Preparation  and  planning  for  inflight  activities. 

1 1 .  Distractions  avoided  or  prioritized. 

12.  Workload  distributed  and  communicated. 

13.  Overall  Workload. 

14.  Overall  TECHNICAL  proficiency. 

1 5.  Overall  CREW  effectiveness 


1 6.  Management  of  abnormal  or  emergency  situation 

17.  Conflict  resolution. 


Aircrew  Coordination  Evaluation  (ACE) 

1.  Thorough  pre-flight  mission  plan  developed. 


2.  Statements/directives  clear,  timely,  relevant,  complete, 
and  verified. 

3.  Inquiry/questioning  practiced. 

4.  Advocacy/assertion  practiced. 

5.  Decisions  communicated  and  acknowledged. 

6.  Actions  communicated  and  acknowledged. 

7.  Crew  self-critique  of  decisions  and  actions. 

8.  Crewmember  actions  mutually  cross  monitored. 

9.  Interpersonal  relationships/group  climate. 

10.  Aircraft,  personnel  and  mission  status  reported. 

1 1 .  Distractions  avoided  or  prioritized. 

12.  Workload  effectively  distributed/redistributed. 

1 3.  Support  information/actions  sought  from  crew. 

14.  Support  Information/actions  offered  by  crew. 

15.  Overall  TECHNICAL  proficiency. 

1 6.  Overall  CREW  effectiveness. 

17.  Overall  workload. 

1 8.  Management  of  abnormal  or  emergency  situation. 

19.  Conflict  Resolution. 


Figure  7..  Comparison  of  the  original  LLC  and  ACE  Checklists  [Adapted  from  R.  Simon,  D.T.  Risser,  E.A. 
Pawlik  and  D.K.  Leedom  (1990).  A  model  for  evaluation  and  training  in  aircrew  coordination  and  cockpit 
resource  management.  In  Proceedings  of  the  Human  Factors  Society  34^^  Annual  Meeting  (pp.  1377-1381). 
Santa  Monica,  CA:  Human  Factors  Society]. 


From  its  creation  until  1991,  the  original  LLC  was  revised  two  times  (Figure  1  shows 
the  items  from  the  unrevised,  original  form),  and  both  revisions  involved  deletion  of  items. 
The  first  revision  deleted  items  13,16,  and  17,  and  the  second  revision  (third  edition)  further 
reduced  the  number  of  observed  behaviors  to  eight  specific  and  two  global  items  (Clothier, 
1991).  The  rating  items  on  the  third  edition  of  the  LLC  were  functionally  the  same  as  the 
FAA  dimensions  of  CRM,  and  used  the  FAA  behavioral  markers  as  references  in  providing  a 
composite  score  for  each  of  the  rating  items  (Law  and  McFadden,  1992).  In  other  words,  the 
LLC,  although  reduced  in  number  of  top-level  dimensions,  required  the  level  of  detail  found 
in  the  FAA’s  behavioral  markers.  Expert  raters  had  to,  at  the  very  least,  be  familiar  with  the 
behavioral  markers  mapped  to  each  LLC  rating  item  to  effectively  provide  a  composite  score 
for  a  dimension.  See  Appendix  A  for  a  full  listing  of  the  FAA  CRM  clusters  and  dimensions, 
and  their  associated  behavioral  markers.  The  latest  version  (version  4)  of  the  LLC 
(Helmreich,  Butler,  Taggart,  and  Wilhelm,  1995)  uses  31  behavioral  markers  to  directly 
evaluate  unique  CRM  behaviors.  See  Appendix  B  for  a  reproduction  of  LLC  version  4. 

The  ACE  Checklist,  like  the  LLC,  has  also  been  revised  from  its  original  form.  This 
revision  involved  the  refinement  of  the  ACE  Checklist  into  13  dimensions  as  shown  in 
Figure  2.  Like  LLC  version  3,  although  the  overall  number  of  dimensions  has  been  reduced, 
the  ACE  Checklist  is  heavily  reliant  upon  further  descriptive  information  to  complete  an 
accurate  evaluation  of  CRM  behaviors.  In  contrast  to  the  LLC  version  3,  however,  the  ACE 
does  not  use  individual  behavioral  markers  (in  short  form)  to  provide  the  necessary 
descriptive  information.  Instead,  the  ACE  requires  the  use  of  a  descriptive  narrative  (about  a 
page  in  length  for  each  rating  item)  which  provides  general  information  about  each  crew 
coordination  basic  quality.  Within  this  narrative  is  also  a  description  of  three  points  on  the 
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seven-point  behaviorally-anchored  rating  scale,  namely  the  very  poor  (1),  acceptable  (4),  and 
superior  (7)  ratings,  which  are  to  be  used  as  anchors. 


AIRCREW  COORDINATION  EVALUATION  (ACE)  CHECKLIST 

For  use  of  this  form,  see  Aircrew  Coordination  Exportable  Evaluation 
Package  for  Army  Aviation. 


CREW  COORDINATION  BASIC  QUALITIES 

RATING 

Establish  and  maintain  flight  team  leadership  and  crew  climate 
(Crew  Climate) 

Premission  planning  and  rehearsal  accomplished  (Plan  Rehearse) 

Application  of  appropriate  decision  making  techniques  (Decision  Tech) 

Prioritize  actions  and  distribute  workload  (Workload) 

Management  of  unexpected  events  (Unexp  Events) 

Statements  and  directives  clear,  timely,  relevant,  complete,  and  verified 
(Info  Xfer) 

Maintenance  of  situational  awareness  (Sit  Aware) 

Decisions  and  actions  communicated  and  acknowledged  (Comm/Ack) 

Supporting  information  and  actions  sought  from  crew  (Info  Sought) 

Crewmember  actions  mutually  cross  monitored  (Cross  Monitor) 

Supporting  information  and  actions  offered  by  crew  (Info  Offered) 

Advocacy  and  assertion  practiced  (Advoc/Assert) 


13  I  Crew-level  after-action  reviews  accomplished  (AAR) _ | _ 

Evaluator’s  Signature: _ 


Notes: 

Consult  the  behavioral  anchored  rating  guidance.  Enter  a  summary  rating  (1,  2  ...  7)  in  the 
rating  block  for  each  Basic  Quality.  Refer  to  the  rating  scale  below. 


RATING  SCALE 


Very  Poor 
1 


Poor 

Marginal 

Acceptable 

Good 

Very  Good 

2 

3 

4 

5 

6 

Superior 

7 


Figure  2,  The  Aircrew  Coordination  Evaluation  (ACE)  Checklist  [Adapted  from 
R.A.  Simon  and  G.N.  Grubb  (1995).  Validation  of  crew  coordination  training  and 
evaluation  methods  for  Army  aviation  (Report  No.  RN-95-45).  Alexandria,  VA:  US 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences]. 


3.2.3  CRM  Survey  Issue:  Number  and  Independence  of  CRM  Dimensions 


Survey  instruments  such  as  the  LLC  and  ACE  Checklist  contain  many  similarities 
with  the  USAF  and  AMC  CRM  dimensions,  and  therefore  could  be  appropriately  modified 
and  used  for  rating  purposes.  It  must  be  emphasized,  however,  that  the  LLC  (through  version 
3)  and  ACE  Checklists  are  highly  reliant  upon  behavioral  markers  and  narrative  description, 
and  are  not  intended  as  stand-alone  checklists  of  CRM  (LLC  version  4  is  a  stand-alone  CRM 
checklist).  Without  immediate  access  to  the  appropriate  behavioral  markers  and  narrative 
description  reference  materials  (used  as  a  guide  for  the  subject-matter  expert  rater),  it  is  likely 
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that  raters  would  misunderstand  various  CRM  dimensions  and  fail  to  distinguish  the 
important  differences.  The  following  results  suggest  this  conclusion. 

Bowers,  Morgan,  and  Salas  (1991,  p.  309)  faced  a  differentiation  among  dimensions 
problem  in  a  study  of  80  helicopter  flight  tasks  (using  a  sample  size  of  14  pilots).  Originally, 
Bowers  et  al.  (1991)  used  seven  dimensions,  shown  at  the  left  side  in  Figure  3. 

These  researchers  found  a  significant  correlation  of  .95  between  assertiveness  and  leadership, 
a  correlation  of  .92  between  adaptation  and  assertiveness,  and  a  correlation  of  .88  between 
mission  analysis  and  decision  making.  Bowers  et  al.  (1991)  note  that  these  correlations  are 


Figures.  Greatest  Contributors  to  Total  Coordination  Demand 
(Bowers,  Morgan,  and  Salas,  1991) 


“sufficiently  high  to  merit  some  concern  about  the  subject’s  ability  to  discriminate  between 
them”  (p.  311).  When  used  for  the  purpose  of  prediction,  there  were  only  three  major 
contributors,  assertiveness,  communication,  and  situation  awareness,  which  accounted  for 
91%  of  the  variance  in  the  estimating  total  coordination  demand. 

The  results  from  this  study  can  be  interpreted  in  two  ways.  First,  it  can  be  argued  that 
a  great  number  of  dimensions  are  not  needed  to  define  CRM.  Indeed,  Bowers  et  al.  (1991) 
state  that  “it  is  possible  that  fewer  than  seven  behavioral  dimensions  may  adequately  define 
the  domain  of  aircrew  coordination”  (p.  311).  In  this  view,  the  simplification  of  the  LLC  and 
ACE  checklists  make  sense;  superfluous  dimensions  have  been  eliminated. 

However,  Bowers  et  al.  (1991)  also  acknowledge  that  “it  is  possible  that  these 
variables  [highly  correlated  dimensions]  are,  in  fact,  conceptually  different  behavioral 
dimensions,  but  that  the  current  sample  did  not  possess  sufficient  understanding  of  the 
differences  among  them,  leading  to  the  high  intercorrelations”  (p.  311).  In  this  view,  it  also 
makes  sense  that  the  LLC  (through  version  3)  and  ACE  Checklists  absolutely  require  the  use 
of  further  clarification  of  dimensions  (either  through  behavioral  markers  or  narrative 
description)  to  provide  the  rater  with  sufficient  understanding  of  the  dimensions.  It  also 
follows  that  version  4  of  the  LLC  directly  provides  the  observer  with  unique,  specific 
behavioral  markers  to  rate.  In  this  manner,  LLC  version  4  eliminates  the  need  to  provide  a 
reference  document  along  with  a  rating  checklist;  each  behavior  is  defined  on  the  LLC  form. 
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Recall  that  CRM  was  originally  developed  to  address  safety  problems  caused  by  poor 
crew  coordination.  CRM  has  since  grown  into  a  much  larger  concept,  becoming  more 
broadly  (and  perhaps  less  succinctly)  defined.  This  presents  major  problems  when 
attempting  to  assess  CRM,  since  an  intended  CRM  assessment  can  easily  become  a  very 
general  assessment  of  crew  performance  rather  than  crew  coordination.  Because  CRM  is 
such  a  broad  concept,  there  is  some  skepticism  within  the  flying  community  that  CRM  can  be 
objectively  measured  (Murray,  Weeks  and  Siem,  1995). 

In  view  of  these  concerns  and  the  results  presented  in  Bowers  et  al.  (1991),  CSERIAC 
recommends  employing  an  assessment  strategy  that  seeks  to  capitalize  on  the  benefits  of  both 
reducing  the  number  of  CRM  dimensions  and  also  more  clearly  defining  each  dimension  in 
behavioral  terms.  This  strategy  narrows  the  number  of  dimensions  defining  CRM  by 
analyzing  the  distinctiveness  of  each  dimension,  and  more  clearly  defines  each  dimension  by 
supplying  clear,  behavioral  markers  to  clarify  the  individual  components  of  each  dimension. 

3.2.4  Linking  CRM  Dimensions  and  CRM  Behaviors 

3.2.4.1  FAA  AC  120-51B  Behavioral  Markers.  The  FAA  (1995)  behavioral 
markers  are  perhaps  the  best  place  to  start  for  the  purpose  of  building  a  robust  CRM  rating 
method  based  on  overt,  observable  performance.  Compared  to  the  other  literature  reviewed 
for  this  report,  the  FAA  (1995)  provides  the  most  comprehensive  listing  of  effective 
performance  markers  associated  with  CRM.  These  markers  (provided  in  Appendix  A)  were 
used  in  the  development  of  LLC  version  4  and  are  believed  to  be  directly  applicable  to  the 
current  need  for  behavioral  markers  for  the  AMC  CRM  ARS  reporting  form. 

More  AMC-specific  needs  for  behavioral  CRM  assessment  have  been  addressed  by 
an  AMC-developed  behavioral  checklist,  the  AMC  CRM  Assessment  Sheet.  It  is  believed 
that  the  best  evaluation  solution  will  be  initiated  by  first  synthesizing  the  behavioral  markers 
from  the  FAA  with  the  AMC  behavioral  markers. 

3.2.4.2  AMC  CRM  Assessment  Sheet.  AMC  has  developed  the  AMC  CRM 
Assessment  Sheet  from  Supplement  1  dimensions  (Table  3).  This  form  uses  ratings  of 
behavioral  observations  (on  a  Likert  scale  of  1-4),  with  1  representing  poor  performance,  2 
representing  minimum  expectations,  3  representing  standard,  and  4  representing  outstanding 
performance.  Each  of  the  six  CRM  core  curriculum  subjects  outlined  by  AMC  is  used  as  a 
heading  in  the  assessment  sheet,  including  Group  Dynamics,  Effective  Communications, 
Assertiveness,  Decision  Making,  Stress  Management,  and  Mishap  Prevention.  A  seventh 
category.  Overall  Observations,  is  used  to  rate  the  crew’s  assessment  of  the  complexity  of  the 
environment,  severity  of  abnormal  systems  operation  and  overall  crew  technical  proficiency. 
The  other  six  CRM  elements  have  from  three  to  eleven  crew  performance  behaviors  rated 
under  each  element.  An  additional,  expansive  comments  section  is  provided  next  to  each  of 
the  items  so  raters  may  provide  additional  comments  not  captured  by  the  behavioral 
observation  rating  scales.  See  Appendix  C  for  an  abbreviated  (comments  section  has  been 
deleted)  view  of  the  AMC  CRM  Assessment  Sheet. 

Some  considerations  should  be  pointed  out  regarding  the  CRM  Assessment  Sheet, 
however.  Several  redundant  behavioral  markers  were  identified.  Redundancy  of  behavioral 
markers  across  dimensions  was  considered  appropriate  in  the  development  of  the  Assessment 
Sheet  according  to  Supplement  1,  section  8.4.4  (USAF,  1995).  However,  this  strategy  will 
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prove  ineffective  in  helping  to  uniquely  define  each  of  the  dimensions  in  terms  of  behaviors 
because  the  same  behavior  is  used  to  represent  two  or  three  different  dimensions. 
Additionally,  the  concepts  of  mishap  prevention  dimension  overlaps  heavily  with  other  API 
36-2243  and  Supplement  1  dimensions.  Both  of  these  problems  will  challenge  the  ability  to 
map  proposed  ARS  form  behaviors  back  to  legitimate  CRM  training  dimensions. 

3.2.4.3  Comparison  of  FAA  and  AMC  Behavioral  Markers.  Because  of  the  need 
for  a  comprehensive  list  of  unique  behavioral  markers,  CSERIAC  compared  the  FAA  AC 
120-5 IB  (FAA,  1995)  and  AMC  CRM  Assessment  Sheet  behaviors  and  developed  a  new  list 
of  unique  CRM  behaviors.  See  Table  4  for  a  listing  and  comparison  of  these  behaviors. 

In  column  one  of  Table  4,  the  FAA  CRM  behavioral  markers  (FAA,  1995)  are  listed. 
For  simplicity,  the  original  structure  of  the  FAA  behavior  listing  has  been  maintained,  and 
the  FAA  CRM  subtopic  headings  are  delineated  in  the  list.  For  instance,  behavior  number  1 
of  the  Briefings  subtopic  in  Table  4  will  also  appear  in  FAA  AC  120-5  IB  (FAA,  1995)  as  the 
first  behavior  of  the  Briefings  subtopic. 

Immediately  following  the  column  1  FAA  behavioral  markers  are  the  AMC  CRM 
Assessment  Sheet  behavioral  markers  (in  column  2).  AMC  behavioral  markers  are  placed 
directly  across  from  the  FAA  behaviors  (in  the  same  row)  that  are  equivalent.  In  certain 
cases,  the  AMC  behavioral  markers  have  the  combined  behaviors  of  several  individual  FAA 
behavioral  markers  included  within  a  single  AMC  rating  item.  In  these  circumstances,  the 
complete  AMC  behavior  is  listed  in  column  2  (next  to  the  individual  FAA  behavior  on  the 
same  row  in  column  1),  but  the  AMC  portion  equivalent  to  the  individual  FAA  behavioral 
marker  is  identified  by  bold  print.  The  AMC  behavioral  markers  are  referenced  to  their 
source  document  (the  AMC  CRM  Assessment  Sheet)  with  a  subject  and  number  code.  For 
instance,  a  code  of  SM4  indicates  Assessment  Sheet  dimension  Stress  Management  and  item 
number  4.  Other  codes  include:  GD  {Group  Dynamics),  AS  (Assertiveness),  EC  (Effective 
Communications),  DM  (Decision  Making),  MP  (Mishap  Prevention),  and  00  (Overall 
Observations).  A  combination  of  two  codes  separated  by  a  comma  (e.g.,  SM4,  MP3) 
indicates  that  the  same  behavior  is  replicated  in  another  AMC  CRM  Assessment  Sheet 
dimension  (in  this  case,  the  overlap  occurs  between  Stress  Management  item  number  4  and 
Mishap  Prevention  item  number  3). 

In  all  cases,  FAA  and  AMC  behavioral  markers  with  shared  properties  appear  in  the 
same  row.  In  certain  cases,  there  is  no  equivalent  AMC  behavior  for  a  particular  FAA 
behavior  (and  vice  versa).  In  these  cases,  a  blank  space  in  the  appropriate  column  indicates 
that  no  equivalent  behavior  was  identified.  In  a  small  number  of  cases,  the  FAA  and  AMC 
behavioral  markers  shared  most  qualities,  but  had  important  differences.  In  these  cases,  the 
similar  behavioral  markers  are  still  included  in  the  same  row,  but  notes  describing  the 
differences  are  added  to  the  items,  appearing  in  (italics  and  parentheses). 

Overall,  the  goal  of  this  comparison  was  to  identify  as  many  unique  CRM-specific 
behavioral  markers  which  could  be  used  for  the  purpose  of  rating  CRM  on  the  AMC  CRM 
ARS  form.  Therefore,  where  the  FAA  and  AMC  behaviors  concur  (on  a  row),  the  selection 
rule  is  to  maintain  a  single  behavioral  marker  for  that  row.  In  cases  where  a  behavioral 
marker  is  unique  to  either  the  FAA  or  AMC  list,  the  selection  rule  is  to  maintain  that  unique 
behavioral  marker  for  the  final,  comprehensive  behavioral  marker  list. 
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Table  4 

Comparison  of  FA  A  AC  120-5  IB  and  AMC  CRM  Assessment  Sheet  Behavioral  Markers 


FAA  CRM  SUBTOPIC:  Briefings 


FAA  AC  1 20-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent 

(1 )  The  briefing  establishes  an  environment  for  open/interactive 
communications  (for  example,  the  captain  calls  for  questions  or 
comments,  answers  questions  directly,  listens  with  patience, 
does  not  interrupt  or  “talk  over,”  does  not  rush  through  the 
briefing,  and  makes  eye  contact  as  appropriate). 

GDI :  Crewmembers  establish  and  maintain  a  team  concept  and 
an  environment  for  open  communications  (I.e.,  crewmembers 
listen  with  patience,  do  not  interrupt  or  “talk  over,”  do  not  rush 
through  the  briefings,  &  make  eye  contact  when  appropriate). 

(2)  The  briefing  is  interactive  and  emphasizes  the  importance  of 
questions,  critique,  and  the  offering  of  information. 

(3)  The  briefing  establishes  a  “team  concept”  (for  example,  the 
captain  uses  “we”  language,  encourages  all  to  participate  and  to 
help  with  the  flight). 

GDI :  Crewmembers  establish  and  maintain  a  team  concept 
and  an  environment  for  open  communications  (i.e.,  crewmembers 
listen  with  patience,  do  not  interrupt  or  “talk  over,”  do  not  rush 
through  the  briefings,  &  make  eye  contact  when  appropriate). 

(4)  The  briefing  covers  pertinent  safety  and  operational  issues. 

(5)  The  briefing  identifies  potential  problems  such  as  weather, 
delays,  and  abnormal  system  operations. 

(6)  The  briefing  provides  guidelines  for  crew  actions;  division  of 
labor  and  crew  workload  is  addressed. 

GD2:  The  entire  crew  participates  in  briefings  as  a  team,  when 
appropriate,  and  the  crew  establishes  guidelines  for 
coordination  between  all  crew  positions.  Crewmembers  brief 
and  update  passengers  when  needed  (i.e.,  mx  delays,  weather, 
etc.). 

(7)  The  briefing  Includes  the  cabin  crew  as  part  of  the  team. 

GD2:  The  entire  crew  participates  in  briefings  as  a  team,  when 
appropriate,  and  the  crew  establishes  guidelines  for  coordination 
between  all  crew  positions.  Crewmembers  brief  and  update 
passengers  when  needed  (i.e.,  mx  delays,  weather,  etc.). 

(8)  The  briefing  sets  expectations  for  handling  deviations  from 
standard  operating  procedures. 

EC1:  The  crew’s  briefings  are  operationally  thorough,  interesting, 
and  address  crew  coordination  while  planning  for  potential 
problems.  The  crew  sets  expectations  on  how  to  handle 
deviations  from  normal  operations. 

(9)  The  briefing  establishes  guidelines  for  the  operation  of 
automated  systems  (for  example,  when  systems  will  be 
disabled;  which  programming  actions  must  be  verbalized  and 
acknowledged). 

MP2:  The  crew  establishes  guidelines  for  the  operation  of 
automated  systems  (i.e.,  when  they  will  disable  systems  and  when 
they  must  verbalize  and  acknowledge  programming  actions). 

(10)  The  briefing  specifies  pilot  flying  and  pilot  not  flying  duties 
and  responsibilities  with  regard  to  automated  systems. 

SM4,  MP3:  The  pilot  team  (crew)  outlines  PF  and  PNF  duties  and 
responsibilities  with  regard  to  automated  systems  (i.e.,  FMS  entry 
and  cross  checking). 

EC1 :  The  crew’s  briefings  are  operationally  thorough,  interesting, 
and  address  crew  coordination  while  planning  for  potential 
problems.  The  crew  sets  expectations  on  how  to  handle 
deviations  from  normal  operating  procedures. 

GD8:  When  appropriate,  crewmembers  take  the  initiative  and  time 
to  share  operational  knowledge  and  experience  (i.e.,  new: 
crewmembers,  routing,  airfields,  situations). 

FAA  CRM  SUBTOPIC:  Inquiry/Advocacy/Assertion 


FAA  AC  1 20-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent 

(1)  Crewmembers  speak  up  and  state  their  information  with 

EC2,  AS2:  Crewmembers  speak  up  and  state  their  information 

appropriate  persistence  until  there  is  some  clear  resolution. 

with  appropriate  persistence,  until  there  is  some  clear  resolution 
and  decision  (i.e.,  effective  advocacy  and  assertion). 

(2)  “Challenge  and  response”  environment  Is  developed. 

(3)  Questions  are  encouraged  and  are  answered  openly  and 
nondefensively. 

(4)  Crewmembers  are  encouraged  to  question  the  actions  and 
decisions  of  others. 

AS1 :  Crewmembers  openly  ask  questions  regarding  crew  actions 
and  decisions  (i.e.,  effective  inquiry). 

1  (5)  Crewmembers  seek  help  from  others  when  necessary.  | 

(6)  Crewmembers  question  status  and  programming  of 
automated  systems  to  confirm  situational  awareness. 

EC4,  MP4:  Crewmembers  periodically  review  and  verify  the  status 
of  aircraft  automated  systems.  (Note:  This  wording  does  not 
capture  the  auestioninp  aspect  of  the  FAA  CRM  wording). 

18 


Table  4  (Continued) 

Comparison  of  FAA  AC  120-5  IB  and  AMC  CRM  Assessment  Sheet  Behavioral  Markers 


FAA  CRM  SUBTOPIC:  Crew  Self-Critique  Regarding 
Decisions  and  Actions 


FAA  AC  1 20-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent  I 

(1)  Critique  occurs  at  appropriate  times,  which  may  be  times  of 
low  or  high  workload. 

(2)  Critique  deals  with  positive  as  well  as  negative  aspects  of 
crew  performance. 

GD5,  EC7:  Crewmembers  provide  (receive)  positive  and 
negative  performance  feedback  at  appropriate  times,  and  the 
atmosphere  creates  a  positive  learning  experience  for  the  entire 
crew — feedback  is  specific,  objective,  based  on  observable 
behavior,  and  given  constructively. 

(3)  Critique  involves  the  whole  crew  interactively.  1 

(4)  Critique  makes  a  positive  learning  experience.  Feedback  is 
specific,  objective,  usable,  and  constructively  given. 

GD5,  EC7:  Crewmembers  provide  (receive)  positive  and 
negative  performance  feedback  at  appropriate  times,  and  the 
atmosphere  creates  a  positive  learning  experience  for  the 
entire  crew — feedback  is  specific,  objective,  based  on 
observabie  behavior,  and  given  constructiveiy. 

(5)  Critique  is  accepted  objectively  and  nondefensively. 

GD6,  ECS:  Crewmembers  accept  performance  feedback 
objectively  and  nondefensively. 

FAA  CRM  SUBTOPIC:  Conflict  Resolution 

1  FAA  AC  120-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent  I 

GD7,  AS3,  SM7:  When  conflicts  arise,  the  crew’s  focus  remains 
on  the  problem  or  situation  at  hand.  Crewmembers  listen  actively 
to  each  others’  ideas  and  opinions  and  admit  mistakes  when 
wrong  (i.e.,  the  crew  resolves  conflict). 

FAA  CRM  SUBTOPIC:  Communications  and 

Decisionmaking 

FAA  AC  1 20-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent 

(1)  Operational  decisions  are  clearly  stated  to  other 
crewmembers. 

EC3,  DM1:  Crewmembers  clearly  state  operational  decisions 
to  other  crewmembers  and  receive  acknowledgment. 
Crewmembers  communicate  the  decisions  to  the  entire  aircrew 
and  others  when  appropriate. 

(2)  Crewmembers  acknowledge  their  understanding  of 
decisions. 

EC3,  DM1 :  Crewmembers  clearly  state  operational  decisions  to 
other  crewmembers  and  receive  acknowledgment. 

Crewmembers  communicate  the  decisions  to  the  entire  aircrew 
and  others  when  appropriate. 

(3)  “Bottom  lines”  for  safety  are  established  and  communicated.  | 

(4)  The  “big  picture”  and  the  game  plan  are  shared  within  the 
team,  including  flight  attendants  and  others  as  appropriate. 

EC3,  DM1 :  Crewmembers  clearly  state  operational  decisions  to 
other  crewmembers  and  receive  acknowledgment. 

Crewmembers  communicate  the  decisions  to  the  entire 
aircrew  and  others  when  appropriate. 

(5)  Crewmembers  are  encouraged  to  state  their  own  ideas, 
opinions,  and  recommendations. 

(6)  Efforts  are  made  to  provide  an  atmosphere  that  invites  open 
and  free  communications. 

(7)  Initial  entries  and  changed  entries  to  automated  systems 
are  verbalized  and  acknowledged. 

EC5:  Crewmembers  verbalize  and  acknowledge  entries  and 
changes  to  automated  systems  parameters. 

002:  The  crew  assesses  the  severity  of  abnormal  systems 
operation  and  other  systems  events  during  the  mission. 
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Table  4  (Continued) 

Comparison  of  FA  A  AC  120-5  IB  and  AMC  CRM  Assessment  Sheet  Behavioral  Markers 


FAA  CRM  SUBTOPIC:  Leadership/Followership/Concern  for 
Tasks 


FAA  AC  1 20-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent  | 

(1)  All  available  resources  are  used  to  accomplish  the  job  at 
hand. 

(2)  Flightdeck  activities  are  coordinated  to  establish  an 

GD4,  DM2:  The  aircraft  commander  coordinates  flightdeck 

acceptable  balance  between  respect  for  authority  and  the 
appropriate  practice  of  assertiveness. 

activities  to  establish  proper  balance  between  command 
authority  and  crewmember  participation.  The  aircraft 
commander  acts  decisively  when  the  situation  requires.  (Note: 

AMC  specifies  this  as  a  responsibility  of  the  aircraft  commander  in 
contrast  to  the  FAA). 

(3)  Actions  are  decisive  when  the  situation  requires. 

GD4,  DM2:  The  aircraft  commander  coordinates  flightdeck 
activities  to  establish  proper  balance  between  command  authority 
and  crewmember  participation.  The  aircraft  commander  acts 
decisively  when  the  situation  requires.  (Note:  AMC  specifies 
this  as  a  responsibility  of  the  aircraft  commander  in  contrast  to  the 
FAA). 

(4)  A  desire  to  achieve  the  most  effective  operation  possible  is 
clearly  demonstrated. 

(5)  The  need  to  adhere  to  standard  operating  practices  is 
recognized. 

(6)  Group  climate  appropriate  to  the  operational  situation  is 
continually  monitored  and  adjusted  (for  example,  social 
conversation  may  occur  during  low  workload,  but  not  high). 

GD3:  Group  climate  matches  the  operational  situation  (i.e., 
presence  or  lack  of  social  conversation).  The  crew  also 
ensures  these  non-operational  factors  to  not  interfere  with 
necessary  tasks. 

(7)  Effects  of  stress  and  fatigue  on  performance  are 
recognized. 

(8)  Time  available  for  the  task  Is  well  managed. 

SMI :  Crewmembers  clearly  communicate  workload  and  task 
distribution  and  receive  acknowledgment  from  other 
crewmembers.  The  crew  allots  adequate  time  to  complete 
tasks. 

(9)  Demands  on  resources  posed  by  operation  of  automated 
systems  are  recognized  and  managed. 

(10)  When  programming  demands  could  reduce  situational 
awareness  or  create  work  overloads,  levels  of  automation 
are  reduced  appropriately. 

SM6:  The  crew  uses  automated  systems  at  optimal  levels  (i.e., 
when  programming  demands  could  reduce  situational  awareness 
and  create  work  overloads,  the  crew  reduces  the  level  of 
automation  or  disengages  automated  systems). 

FAA  CRM  SUBTOPIC:  Interpersonal  Relationships/Group 

Climate _ 

FAA  AC  1 20-51 B  Behavioral  Markers _ AMC  CRM  Assessment  Sheet  Equivalent _ 

(1)  Crewmembers  remain  calm  under  stressful  conditions.  DM4,  SM1 1 :  The  crew  remains  calm  under  stress. 


(2)  Crewmembers  show  sensitivity  and  ability  to  adapt  to  the 

personalities  of  others. _ 

(3)  Crewmembers  recognize  symptoms  of  psychological  stress 

and  fatigue  in  self  and  in  others  (for  example,  recognizes  when 
he/she  is  experiencing  “tunnel  vision”  and  seeks  help  from  the 
team;  or  notes  when  a  crewmember  is  not  communicating  and 
draws  him/her  back  into  the  team). _ 

(4)  “Tone”  in  the  cockpit  is  friendly,  relaxed,  and  supportive. 

(5)  During  times  of  low  communication,  crewmembers  check  In 

with  others  to  see  how  they  are  doing.  (Note:  AMC  specifies 
that  crewmembers  are  to  check  in  with  others  during  times  of 
high  and  low  workload). _ 
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Table  4  (Continued) 

Comparison  of  FA  A  AC  120-5  IB  and  AMC  CRM  Assessment  Sheet  Behavioral  Markers 


FAA  CRM  SUBTOPIC:  Preparatlon/PlanningA/igilance 


FAA  AC  1 20-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent  | 

(1)  Demonstrating  and  expressing  situational  awareness;  for 
example,  the  “model”  of  what  is  happening  is  shared  within  the 
crew. 

(2)  Active  monitoring  of  all  Instruments  and  communications  and 
sharing  relevant  information  with  the  rest  of  the  crew. 

001 :  The  crew  consistently  assesses  the  complexity  of  the 
operating  environment  (WX,  ATC,  traffic,  threat). 

(3)  Monitoring  weather  and  traffic  and  sharing  relevant  Information 
with  the  rest  of  the  crew. 

001:  The  crew  consistently  assesses  the  complexity  of  the 
operating  environment  (WX,  ATC,  traffic,  threat). 

(4)  Avoiding  “tunnel  vision”  caused  by  stress;  for  example,  stating 
or  asking  for  the  “big  picture.” 

(5)  Being  aware  of  factors  such  as  stress  that  can  degrade 
vigilance  and  watching  for  performance  degradation  in  other 
crewmembers. 

(6)  Staying  “ahead  of  the  curve”  in  preparing  for  planned  situations 
or  contingencies. 

DM3,  SM3:  The  crew  prepares  for  expected  or  contingency 
situations  including  approaches,  weather,  etc.  (i.e.,  the  crew 
stays  ahead  of  the  power  curve). 

(7)  Ensuring  that  cockpit/cabin  crewmembers  are  aware  of  plans. 

(8)  Including  all  appropriate  crewmembers  in  the  planning  process. 

(9)  Allowing  enough  time  before  maneuvers  for  programming  of  the 
flight  management  computer. 

EC6,  SMS,  MP6:  Crewmembers  allow  (plan  for)  sufficient 
time  for  programming  of  flight  management  computers  prior  to 
maneuvers. 

(10)  Ensuring  that  all  crewmembers  are  aware  of  initial  entries  and 
changed  entries  in  the  flight  management  system. 

ECS,  MPS:  Crewmembers  verbalize  and  acknowledge  entries 
and  changes  to  automated  systems  parameters. 

SM9:  The  crew’s  actions  do  not  create  self-imposed  stress 
and  additional  workload  (i.e.,  a  late  descent  due  to  lack  of 
situational  awareness/planning). 

FAA  CRM  SUBTOPIC:  Workload  Distributed/Distractions 
Avoided 


FAA  AC  1 20-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent 

(1)  Crewmembers  speak  up  when  they  recognize  work  overloads 
in  themselves  or  in  others. 

SM10:  Crewmembers  recognize  and  report  when  their  duties 
or  the  duties  they  observe  others  performing  cause  overload. 

(2)  Tasks  are  distributed  In  ways  that  maximize  efficiency.  | 

(3)  Workload  distribution  is  clearly  communicated  and 
acknowledged. 

SM1:  Crewmembers  clearly  communicate  workload  and 
task  distribution  and  receive  acknowledgment  from  other 
crewmembers.  The  crew  allots  adequate  time  to  complete 
tasks. 

(4)  Non-operational  factors  such  as  social  interaction  are  not 
allowed  to  interfere  with  duties. 

GD3:  Group  climate  matches  the  operational  situation  (i.e., 
presence  or  lack  of  social  conversation).  The  crew  also 
ensures  these  non-operational  factors  do  not  interfere 
with  necessary  tasks. 

(5)  Task  priorities  are  clearly  communicated.  | 

(6)  Secondary  operational  tasks  (for  example,  dealing  with 
passenger  needs  and  communications  with  company)  are  prioritized 
so  as  to  allow  sufficient  resources  for  primary  flight  duties. 

SM2:  The  crew  prioritizes  secondary  operational  tasks  (i.e., 
dealing  with  passenger  needs,  command  post 
communications)  to  retain  sufficient  resources  to  deal 
effectively  with  primary  task  duties. 

(7)  Potential  distractions  posed  by  automated  systems  are 
anticipated,  and  appropriate  preventive  action  is  taken,  including 
reducing  or  disengaging  automated  features  as  appropriate. 

MP1 :  Crewmembers  check-in  with  each  other  during  times  of 
high  and  low  workload  to  maintain  situational  awareness  and 
to  remain  alert.  (Note:  The  FAA  states  that  crewmembers 
check  in  with  each  other  during  times  of  low  communication). 

FAA  CRM  SUBTOPIC:  Individual  Factors/Stress  Reduction 

FAA  AC  120-51 B  Behavioral  Markers 

AMC  CRM  Assessment  Sheet  Equivalent 

SM8:  During  long  duty  periods,  crewmembers  are  pro-active 
in  remaining  alert,  and  plan  and  use  fatigue  countermeasures. 
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CSERJAC  found  considerable  agreement  between  the  two  lists  of  behavioral  markers, 
adding  support  to  the  belief  that  an  integration  of  the  two  lists  could  provide  an  appropriate 
master  behavioral  marker  list  for  the  AMC  CRM  ARS  form.  In  eertain  cases,  the  AMC 
behavioral  markers  appeared  to  combine  several  of  the  FAA  behavioral  markers  into  a  single 
rating  item.  CSERJAC  elected  to  use  the  original  FAA  behavioral  markers  in  these  cases. 
This  rationale  was  taken  in  order  to  rate  individual  behaviors  rather  than  eombinations  of 
behaviors.  The  comparison  also  emphasized  that  several  of  the  AMC  behavioral  markers 
overlap  with  each  other  in  different  AMC  CRM  dimensions,  making  them  difficult  to  map  to 
a  specific  dimension.  In  several  instances,  however,  each  of  the  FAA  and  the  AMC  lists 
provided  behavioral  markers  not  found  in  the  other  of  the  two  lists. 

Overall,  this  comparison  began  with  a  total  of  60  FAA  and  44  AMC  CRM  behavioral 
markers.  Removing  the  overlapping  effects  between  the  lists  of  behavioral  markers,  (and 
retaining  the  unique  qualities  of  each  behavioral  marker)  CSERIAC  developed  a  final  list  of 
67  behavioral  markers  that  may  be  used  as  a  guide  in  the  development  of  a  revised  AMC 
CRM  ARS  form.  This  list  is  provided  in  Table  5,  in  conjunction  with  a  comparison  of  the 
current  AMC  CRM  ARS  form  12  CRM  factors  (described  below). 

3.2.4.4  Comparison  of  FAA  and  AMC  Behavioral  Markers  with  ARS  Form. 
Upon  integrating  the  FAA  and  AMC  Assessment  Sheet  behavioral  markers,  a  comparison 
with  the  current  version  of  the  AMC  CRM  ARS  form  (see  Appendix  D)  was  performed. 

This  comparison  (Table  5)  emphasized  the  weaknesses  of  the  current  ARS  form,  and  also 
emphasized  the  strengths  of  using  behavioral  markers  for  the  purpose  of  rating. 

As  presently  configured,  the  CRM  ARS  reporting  form  lists  12  factors  that  contribute 
to  problems  with  CRM.  These  factors  include  Interruption  in  a  planned  activity,  Non¬ 
standard/misunderstood  communications,  Proficiency,  Misleading/erroneous  guidance  or 
manuals.  High  workload.  Complacency,  Inadequate  planning.  Crew  coordination. 

Equipment  failure.  Visual  illusion/spatial  disorientation.  Insufficient  training,  and  Stress  or 
anxiety.  In  several  instances,  these  12  factors  do  address  aspects  of  CRM.  However,  the 
factors  lack  the  specifieity  of  behavioral  markers  and  the  definition  of  the  CRM  dimensions. 
As  shown  in  Table  5,  the  factors  that  do  relate  to  behavioral  markers  are  generally  too  broad 
to  assess  the  level  of  detail  found  in  the  behavioral  markers.  In  many  cases,  several  factors 
overlap  a  single  marker,  bringing  associated  problems  with  differentiation  between 
dimensions. 

For  instance,  the  Crew  coordination  factor  cam  be  associated  with  39  of  the  67 
combined  behavioral  markers  (this  fact  is  not  surprising  considering  that  the  entire  CRM 
concept  is  focused  on  crew  coordination).  The  association  of  Crew  coordination  with  a  high 
number  of  behavioral  markers  raises  concerns  with  the  usefulness  of  this  factor;  it  is  likely 
that  it  will  not  accurately  pinpoint  problems  with  current  USAF  CRM  behaviors.  Also,  in  23 
of  the  39  times  that  Crew  coordination  mapped  to  behavioral  markers,  other  AMC  CRM 
ARS  form  factors  also  mapped  to  these  same  behavioral  markers.  This  problem  suggests  that 
the  12  factors  on  the  current  ARS  form  may  not  be  differentiated  enough  from  each  other. 

As  expressed  earlier  in  this  report,  without  behavioral  markers  or  significant  descriptive 
information,  crewmembers  may  not  be  able  to  differentiate  behaviors  into  the  correct 
dimension  (or  in  this  case,  factor). 
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Tables 

Combined  Behavioral  Marker  List  Comparison  with  Present  AMC  CRM  ARS  12  Factors 


Integrated  List  of  FAA  and  USAF  Behavioral  Markers 

AMC  CRM  ARS  Comparison 

(1)  The  briefing  establishes  an  environment  for  open/interactive  communications  (for 

example,  the  captain  calls  for  questions  or  comments,  answers  questions  directly,  listens 
with  patience,  does  not  interrupt  or  “talk  over,”  does  not  rush  through  the  briefing,  and 
makes  eye  contact  as  appropriate). 

Crew  coordination.  Non-standard/ 
misunderstood  communications 

(2)  The  briefing  is  interactive  and  emphasizes  the  importance  of  questions,  critique,  and  the 
offering  of  information. 

Crew  coordination 

(3)  The  briefing  establishes  a  “team  concept"  (for  example,  the  captain  uses  “we”  language, 
encourages  all  to  participate  and  to  help  with  the  flight). 

Crew  coordination 

(4)  The  briefing  covers  pertinent  safety  and  operational  issues. 

Inadequate  planning 

(5)  The  briefing  identifies  potential  problems  such  as  weather,  delays,  and  abnormal  system 
operations. 

Inadequate  planning 

(6)  The  briefing  provides  guidelines  for  crew  actions;  division  of  labor  and  crew  workload  is 
addressed. 

Inadequate  planning,  Crew 
coordination.  High  workload 

(7)  The  briefing  includes  the  cabin  crew  as  part  of  the  team. 

Crew  Coordination 

(8)  The  briefing  sets  expectations  for  handling  deviations  from  standard  operating 
procedures. 

Inadequate  planning. 

Interruption  In  a  planned  activity 

(9)  The  briefing  establishes  guidelines  for  the  operation  of  automated  systems  (for  example, 
when  systems  will  be  disabled;  which  programming  actions  must  be  verbalized  and 
acknowledged). 

Crew  coordination,  Inadequate  planning 

(1 0)  The  briefing  specifies  pilot  flying  and  pilot  not  flying  duties  and  responsibilities  with  regard 
to  automated  systems. 

Crew  coordination,  Inadequate  planning 

(11)  EC1 :  The  crew’s  briefings  are  operationally  thorough,  interesting,  and  address  crew 
coordination  while  planning  for  potential  problems.  The  crew  sets  expectations  on  how  to 
handle  deviations  from  normal  operating  procedures. 

Crew  coordination.  Inadequate  planning 

(12)  GD8:  When  appropriate,  crewmembers  take  the  initiative  and  time  to  share  operational 
knowledge  and  experience  (i.e.,  new:  crewmembers,  routing,  airfields,  situations). 

Crew  coordination 

(13)  Crewmembers  speak  up  and  state  their  information  with  appropriate  persistence  until 
there  is  some  clear  resolution. 

Non-standard/  misunderstood 
communications 

(14)  “Challenge  and  response”  environment  is  developed. 

Complacency 

(15)  Questions  are  encouraged  and  are  answered  openly  and  nondefensively. 

(16)  Crewmembers  are  encouraged  to  question  the  actions  and  decisions  of  others. 

(17)  Crewmembers  seek  help  from  others  when  necessary. 

Crew  coordination.  Complacency 

(18)  Crewmembers  question  status  and  programming  of  automated  systems  to  confirm  SA. 

Crew  coordination,  Complacency 

(19)  Critique  occurs  at  appropriate  times,  which  may  be  times  of  low  or  high  workload. 

(20)  Critique  deals  with  positive  as  well  as  negative  aspects  of  crew  performance. 

(21)  Critique  involves  the  whole  crew  interactively. 

Crew  coordination 

(22)  Critique  makes  a  positive  learning  experience.  Feedback  is  specific,  objective,  usable, 
and  constructively  given. 

Crew  coordination.  Non-standard/ 
misunderstood  communications 

1  (23)  Critique  is  accepted  objectively  and  nondefensively^  ~  I 

(24)  GD7,  AS3,  SM7:  When  conflicts  arise,  the  crew’s  focus  remains  on  the  problem  or 
situation  at  hand.  Crewmembers  listen  actively  to  each  others’  ideas  and  opinions  and 
admit  mistakes  when  wrong  (i.e.,  the  crew  resolves  conflict). 

Crew  coordination 

(25)  Operational  decisions  are  clearly  stated  to  other  crewmembers. 

Non-standard/  misunderstood 
communications,  Crew  coordination 

(26)  Crewmembers  acknowledge  their  understanding  of  decisions. 

Non-standard/  misunderstood 
communications.  Crew  coordination 

(27)  “Bottom  lines”  for  safety  are  established  and  communicated. 

Crew  coordination 

(28)  The  “big  picture”  and  the  game  plan  are  shared  within  the  team,  including  flight 
attendants  and  others  as  appropriate. 

Crew  coordination.  Non-standard/ 
misunderstood  communications 

(29)  Crewmembers  are  encouraged  to  state  their  own  ideas,  opinions,  and  recommendations. 

Crew  coordination 

(30)  Efforts  are  made  to  provide  an  atmosphere  that  invites  open  and  free  communications. 

Crew  coordination 

(31)  Initial  entries  and  changed  entries  to  automated  systems  are  verbalized  and 
acknowledged. 

Crew  coordination 

(32)  002:  The  crew  assesses  the  severity  of  abnormal  systems  operation  and  other  systems 
events  during  the  mission. 

Crew  coordination 

(33)  All  available  resources  are  used  to  accomplish  the  job  at  hand. 

Crew  coordination.  Complacency 

(34)  Flightdeck  activities  are  coordinated  to  establish  an  acceptable  balance  between  respect 
for  authority  and  the  appropriate  practice  of  assertiveness. 

Crew  coordination 

23 


Tables  (Continued) 

Combined  Behavioral  Marker  List  Comparison  with  Present  AMC  CRM  ARS  12  Factors 


Integrated  List  of  FAA  and  USAF  Behavioral  Markers 

AMC  CRM  ARS  Comparison 

(35)  Actions  are  decisive  when  the  situation  requires. 

Complacency 

(36)  A  desire  to  achieve  the  most  effective  operation  possible  is  cleariv  demonstrated. 

Complacency 

(37)  The  need  to  adhere  to  standard  operating  practices  is  recognized. 

Complacency 

(38)  Group  climate  appropriate  to  the  operational  situation  is  continually  monitored  and 
adjusted  (for  example,  social  conversation  may  occur  during  low  workload,  but  not  high) 

Crew  coordination.  Complacency 

(39)  Effects  of  stress  and  fatigue  on  performance  are  recognized. 

Stress  or  anxiety 

(40)  Time  available  for  the  task  is  well  managed. 

High  workload,  Complacency 

(41)  Demands  on  resources  posed  by  operation  of  automated  systems  are  recognized  and 
managed. 

High  workload,  Stress  or  anxiety 

(42)  When  programming  demands  could  reduce  situational  awareness  or  create  work 
overloads,  levels  of  automation  are  reduced  appropriately. 

High  workload.  Stress  or  anxiety 

(43)  Crewmembers  remain  calm  under  stressful  conditions. 

Stress  or  anxiety 

(44)  Crewmembers  show  sensitivity  and  ability  to  adapt  to  the  personaiities  of  others. 

Crew  coordination 

(45)  Crewmembers  recognize  symptoms  of  psychological  stress  and  fatigue  in  self  and  in 
others  (for  example,  recognizes  when  he/she  is  experiencing  “tunnel  vision”  and  seeks 
help  from  the  team;  or  notes  when  a  crewmember  is  not  communicating  and  draws 
him/her  back  into  the  team). 

Stress  or  anxiety.  Visual  illusion/  spatial 
disorientation 

(46)  “Tone”  in  the  cockpit  is  friendly,  relaxed,  and  supportive. 

Crew  coordination 

(47)  During  times  of  low  communication,  crewmembers  check  in  with  others  to  see  how  they 
are  doing.  (Note:  AMC  specifies  that  crewmembers  are  to  check  in  with  others  during 
times  of  hiph  and  low  workload). 

Non-standard/  misunderstood 
communications.  Crew  coordination, 
Complacency 

(48)  Demonstrating  and  expressing  situational  awareness;  for  example,  the  “model”  of  what  is 
happening  is  shared  within  the  crew. 

Visual  illusion/  spatial  disorientation 

(49)  Active  monitoring  of  all  instruments  and  communications  and  sharing  relevant  information 
with  the  rest  of  the  crew. 

Crew  coordination 

(50)  Monitoring  weather  and  traffic  and  sharing  relevant  information  with  the  rest  of  the  crew. 

Crew  coordination 

(51)  Avoiding  “tunnel  vision”  caused  by  stress;  for  example,  stating  or  asking  for  the  “big 
picture.” 

Visual  illusion/  spatial  disorientation 

(52)  Being  aware  of  factors  such  as  stress  that  can  degrade  vigilance  and  watching  for 
performance  degradation  in  other  crewmembers. 

Stress  or  anxiety.  Crew  coordination 

(53)  Staying  “ahead  of  the  curve"  in  preparing  for  planned  situations  or  continaencies. 

Inadequate  Planning,  High  Workload 

(54)  Ensuring  that  cockpit  and  cabin  crewmembers  are  aware  of  plans. 

Crew  coordination.  Inadequate  planning 

(55)  Including  all  appropriate  crewmembers  in  the  planning  process. 

Crew  coordination,  Inadequate  planning 

(56)  Allowing  enough  time  before  maneuvers  for  programming  of  the  flight  management 
computer. 

Inadequate  planning.  High  workload 

(57)  Ensuring  that  ail  crewmembers  are  aware  of  initial  entries  and  changed  entries  in  the 
flight  management  system. 

Crew  coordination 

(58)  SM9:  The  crew’s  actions  do  not  create  self-imposed  stress  and  additional  workload  (i.e., 
a  late  descent  due  to  lack  of  situational  awareness/planning). 

High  workload.  Stress  or  anxiety 

(59)  Crewmembers  speak  up  when  they  recognize  work  overloads  in  themselves  or  in  others 

High  workload.  Stress  or  anxiety 

(60)  Tasks  are  distributed  in  ways  that  maximize  efficiency. 

Crew  coordination,  High  workload 

(61)  Workload  distribution  is  clearly  communicated  and  acknowledged. 

Crew  coordination,  High  workload 

(62)  Non-operational  factors  such  as  social  interaction  are  not  allowed  to  interfere  with  duties. 

Crew  coordination.  Interruption  in  a 
planned  activity 

(63)  Task  priorities  are  clearly  communicated. 

Non-standard/  misunderstood 
communications.  Inadequate  planning 

(64)  Secondary  operational  tasks  (for  example,  dealing  with  passenger  needs  and 

communications  with  company)  are  prioritized  so  as  to  allow  sufficient  resources  for 
primary  flight  duties. 

Inadequate  planning,  Crew 
coordination,  Interruption  in  a  planned 
activity,  High  workload 

(65)  Potential  distractions  posed  by  automated  systems  are  anticipated,  and  appropriate 
preventive  action  is  taken,  inciuding  reducing  or  disengaging  automated  features  as 
appropriate. 

Inadequate  planning,  Interruption  In  a 
planned  activity 

(66)  MPI :  Crewmembers  check-in  with  each  other  during  times  of  high  and  low  workload  to 
maintain  situational  awareness  and  to  remain  alert.  (Note:  The  FAA  states  that 
crewmembers  check  in  with  each  other  during  times  of  iow  communication). 

High  workload.  Stress  or  anxiety, 
Complacency,  Crew  Coordination 

(67)  SMS:  During  long  duty  periods,  crewmembers  are  pro-active  in  remaining  alert,  and  plan 
and  use  fatigue  countermeasures. 

Complacency 
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Overall,  8  of  the  1 2  factors  from  the  present  AMC  CRM  ARS  form  were  found  to  be 
related  to  the  comprehensive  list  of  behavioral  markers.  As  with  the  Crew  coordination 
factor,  the  seven  other  factors  overlap  with  each  other  and  a  number  of  the  behavioral 
markers.  In  other  cases,  there  are  factors  on  the  present  ARS  that  do  not  relate  to  any  aspect 
of  CRM.  This  is  not  to  say  that  these  factors  are  not  useful;  in  some  cases,  they  provide 
queries  for  experience  or  context  information  surrounding  a  particular  incident.  These  factors 
could  be  useful  in  more  accurately  describing  a  CRM  event,  but  should  not  be  categorized  as 
CRM  dimensions. 

In  all  of  the  12  factors  on  the  present  AMC  CRM  ARS  form,  weaknesses  exist  in 
issues  of  definition,  differentiation,  and  utility.  Consider  the  following  recommendations  for 
refinement  of  the  12  current  ARS  factors.  Table  6  provides  the  factors,  their  challenges,  and 
recommendations  for  improvement.  Refer  to  Table  5  for  a  depiction  of  the  extent  of  overlap 
individual  factors  share  with  each  other  and  different  behavioral  markers.  CSERIAC 
recommends  that  the  information  originally  addressed  by  the  12  factors  be  gathered  in  three 
specific  sections:  crew  background,  event  context,  and  CRM  behaviors.  These  sections  are 
described  in  more  detail  in  section  3.3  of  this  report. 


Table  6 

Current  AMC  CRM  ARS  Form  Factors,  Challenges,  and  Recommendations 


AMC  ARS  Form  Factor 

Challenge 

Recommendation 

1)  Interruption  in  a 
planned  activity 

Borderline  CRM-specific  variable,  too  broad, 
cuts  across  many  CRM  behaviors. 

Reassess  factor  as  a  CRM  variable, 
use  specific  behavioral  markers. 

2)  Non-standard/ 
misunderstood 
communications 

Too  broad,  cuts  across  many  CRM 
behaviors. 

Maintain  as  CRM  variable,  but  use 
specific  behavioral  markers. 

3)  Proficiency 

Not  a  CRM-specific  variable. 

Place  in  crew  background  section. 
Include  general  proficiency  and  CRM 
proficiency  in  this  section. 

4)  Misleading/erroneous 
guidance  or  manuals 

Not  a  CRM-specific  variable. 

Place  in  event  context  section. 

5)  High  workload 

Too  broad,  cuts  across  many  CRM 
behaviors. 

Maintain  as  CRM  variable,  but  use 
specific  behavioral  markers. 

6)  Complacency 

Borderline  CRM-specific  variable,  too  broad, 
cuts  across  many  CRM  behaviors. 

Reassess  factor  as  a  CRM  variable, 
use  specific  behavioral  markers. 

7)  Inadequate  planning 

Too  broad,  cuts  across  many  CRM 
behaviors. 

Maintain  as  CRM  variable,  but  use 
specific  behavioral  markers. 

8)  Crew  coordination 

Too  broad,  cuts  across  many  CRM 
behaviors. 

Maintain  as  CRM  variable,  but  use 
specific  behavioral  markers. 

9)  Equipment  failure 

Not  a  CRM-specific  variable. 

Place  in  event  context  section. 

10)  Visual  illusion/ spatial 
disorientation 

Borderline  CRM-specific  variable,  too  broad, 
cuts  across  many  CRM  behaviors. 

Reassess  factor  as  a  CRM  variable, 
use  specific  behavioral  markers. 

11)  Insufficient  training 

Not  a  CRM-specific  variable. 

Place  in  crew  background  section. 

12)  Stress  or  anxiety 

Too  broad,  cuts  across  many  CRM 
behaviors. 

Maintain  as  CRM  variable,  but  use 
specific  behavioral  markers. 

Central  to  the  purpose  of  this  report  is  the  issue  of  appropriate  assessment  of  CRM 
dimensions.  Since  the  current  AMC  CRM  ARS  form  factors  do  not  effectively  assess  CRM, 
a  new  listing  of  appropriate  CRM  rating  items  must  be  developed.  The  previous  discussion 
and  comparison  of  FAA  and  AMC  CRM  behavioral  markers  provided  a  comprehensive  list 
of  behavioral  markers  that  may  be  used  in  rating  CRM.  This  behavioral  marker  list  is 
believed  to  provide  a  basis  for  the  further  refinement  of  the  AMC  CRM  ARS  reporting  form. 
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3.2.4.5  Narrowing  the  CRM  Dimensions  and  Remapping  Behavioral  Markers. 

In  their  present  form,  the  67  behavioral  markers  identified  in  this  report  represent  a  list  that 
defines  CRM  as  a  whole.  These  behavioral  markers  could  be  incorporated  into  a  revised 
AMC  CRM  ARS  reporting  form.  Crewmembers  could  then  provide  ratings  on  specific  CRM 
behaviors  that  were  observed  the  operational  environment.  Having  this  information  alone 
would  provide  significant  insight  into  behaviors  believed  to  compose  CRM  as  a  whole. 

As  an  added  benefit,  the  67  behavioral  markers  may  also  be  mapped  to  the  Air  Force 
and  AMC  CRM  dimensions  identified  earlier  in  Table  3.  Such  a  categorization  would 
provide  specific,  observable  behaviors  for  each  of  the  applicable  CRM  dimensions,  and 
hence,  a  link  back  to  the  core  curriculum.  Such  a  mapping  is  valuable  because  recurring 

problems  exposed  by  the  ARS  database  may  be  traced  back  to  specific  CRM  core  curriculum 
dimensions. 

Since  behavioral  markers  are  believed  to  provide  the  best  basis  for  evaluating  CRM 
behaviors,  CSERIAC  undertook  an  effort  to  identify  possible  behavioral  markers  for  each  of 
the  USAF  CRM  dimensions  described  in  Table  3  (by  using  dimension  definitions),  including 
mission,  planning,  review,  and  critique  strategies,  assertiveness  training,  group  dynamics, 
effective  communications,  stress  awareness  and  management,  workload  management, 
situation  awareness,  and  decision  making.  Human  performance  and  concepts  of  mishap 
prevention  were  intentionally  excluded  from  this  list  because,  (1)  these  areas  overlap  other 
distinct  CRM  concepts  extensively,  and  (2)  there  are  no  behavioral  markers  that  distinctly 
and  uniquely  support  these  constructs.  Note  that  the  heading  overall  observations  is 
excluded  as  well,  as  it  is  not  a  CRM-specific  dimension.  The  results  of  this  categorization  of 
behavioral  markers  into  dimensions  are  provided  in  Table  7. 

Table  7  provides  the  applicable  USAF  CRM  dimensions  in  bold  print  above  the 
associated  behavioral  markers.  Each  behavioral  marker  may  be  traced  back  to  the  combined 
FAA  and  AMC  behavioral  marker  list  (Table  5)  through  the  number  in  bold  and  parentheses 
after  the  marker.  Notice  that  mission,  planning,  review  and  critique  strategies,  group 
dynamics,  and  workload  management  are  all  well  represented  by  behavioral  markers;  each 
dimension  has  more  than  1 0  behaviors  associated  with  it.  Assertiveness  training,  effective 
communications,  and  stress  awareness  and  management  are  fairly  well  represented  with 
between  five  and  seven  behaviors  each,  while  decision  making  and  situational  awareness 
have  four  and  three  associated  behavioral  markers  respectively.  This  list  is  by  no  means 
meant  to  be  the  final  word  in  behavioral  markers  for  each  dimension.  Some  behavioral 
markers  may  need  to  be  added  or  deleted  from  dimensions,  depending  on  the  goals  and  needs 
of  AMC.  The  categorization  does,  however,  provide  a  sound  strategy  for  collecting  CRM 
data  that  is  traceable  to  CRM  training.  Therefore,  when  problems  surface  in  a  particular 
behavior,  that  behavior  is  linked  to  its  dimension  and  core  curriculum  training  course.  From 
this  link,  improvements  to  the  appropriate  CRM  training  course  may  be  made. 
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3.3  Further  Suggestions  for  Present  AMC  CRM  ARS  Form  Improvement 

To  this  point,  the  focus  has  been  placed  on  the  CRM  construct  and  its  definition, 
dimensions,  and  appropriate  rating  methodologies.  This  focus  has  been  taken  because  of  the 
primary  importance  that  CRM  assessment  assumes  in  the  AMC  CRM  ARS  reporting  form. 
Other  issues  central  to  the  refinement  of  the  form  should  be  considered  as  well.  The  present 
section  presents  suggestions  and  structure  that  may  be  applied  to  the  AMC  CRM  ARS  form 
revision.  Based  on  these  suggestions,  an  evaluation  copy  of  a  possible  AMC  CRM  ARS 
form  is  provided  in  Appendix  F.  Note  that  this  form  is  presented  for  comment  and 
refinement  only;  it  does  not  represent  a  final  ARS  form. 

3.3.1  Neutral  Terminology 

AMC  may  want  to  consider  replacing  the  term  incident  in  the  introduction  with  a 
more  neutral  or  less  heavily  loaded  word,  dissociating  the  word  incident  from  CRM.  In  the 
vernacular,  the  term  incident  has  been  used  by  the  Air  Force,  the  FAA,  and  the  NTSB  to 
connote  a  principally  negative  outcome  that  was  short  of  an  accident  by  some  measure  (loss 
of  life,  loss  of  equipment,  loss  of  operational  capacity,  and  so  forth).  Because  the  CRM  ARS 
has  been  designed  to  gather  both  positive  and  negative  examples  of  CRM,  it  is  necessary  that 
the  terminology  in  the  form  reflect  this.  Event  and  situation  are  probably  the  more  desired 
terms  to  use. 

3.3.2  Contact  Information 

It  is  appropriate  to  ask  for  contact  information.  This  information  would  allow  the 
analyst  reviewing  a  CRM  event  to  contact  the  reporter  to  gain  a  better  understanding  of  the 
event.  See  Appendix  E  for  a  view  of  the  NASA  Aviation  Safety  Reporting  System  (ASRS) 
form  contact  information  section.  NASA  ASRS  has  never  breached  anonymity  in  collecting 
this  information  (Drew,  1995).  Contact  information  simply  includes  the  reporter’s  name, 
address,  and  telephone  number,  and  is  used  only  for  the  purpose  of  gaining  additional  insight 
into  a  particular  CRM  event.  The  contact  information  could  (as  with  the  NASA  ASRS)  be 
placed  on  a  section  of  the  final  ARS  form  which  may  be  returned  to  the  reporter  as  an 
additional  assurance  of  anonymity.  Anonymity  may  be  guaranteed  by  not  archiving  the 
contact  data. 

3.3.3  Crew  Background 

The  AMC  CRM  ARS  form  is  intended  to  be  used  to  assess  and  improve  the  training 
system.  Hence,  it  makes  sense  to  ask  questions  regarding  the  particular  respondent’s  past 
background  both  with  general  proficiency  and  with  CRM  (i.e.,  what  CRM  training  has  the 
respondent  taken  part  in?).  The  selections  on  the  AMC  CRM  ARS  form  should  be 
representative  of  CRM  training  methods  and  media  used  in  USAF  and  AMC  courses.  In  this 
manner,  it  may  be  foimd  that  particular  types  of  training  (or  lack  of  them)  are  more  highly 
correlated  with  certain  accidents  (or  augmented  safety)  than  others.  From  these  statistics, 
future  course  content  may  be  tailored  to  improve  training,  and  ultimately,  safety.  In  tailoring 
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the  form,  AMC  may  want  to  exclude  references  to  training  that  is  mandatory  and  consistent 
throughout  AMC,  the  reserve,  and  guard  participants  (i.e.,  eliminate  the  need  to  encode 
information  that  will  be  common  to  all  reporting,  information  that  can  not  provide  any 
discriminative  power  during  subsequent  analysis  efforts). 

Additionally,  general  crew  background  information  may  also  be  desired.  Such 
information  could  include  rank,  ratings,  crew  position  during  CRM  event,  and  general 
proficiency.  See  Appendix  E  for  the  NASA  ASRS  organization  of  crew  background 
information.  Items  originally  appearing  in  the  12  factors  of  the  original  AMC  CRM  ARS 
form  that  deal  with  crew  background  issues  (such  as  proficiency)  could  be  placed  in  this 
section. 

3.3.4  Event  Context 

It  is  important  to  ascertain  the  context  of  the  reported  situation,  including 
environment,  conflicts,  weather,  etc.  The  NASA  ASRS  (Appendix  E)  provides  an  excellent 
example  of  the  kinds  of  items  which  the  context  section  could  contain.  AMC  may  want  to 
tailor  this  section  based  on  what  AMC,  the  reserve,  and  the  guard  crews  are  likely  to 
experience.  This  may  involve  adding  military-specific  checklist  items.  Items  originally 
appearing  in  the  12  factors  of  the  original  AMC  CRM  ARS  form  that  deal  with  event  context 
issues  (such  as  equipment  failure)  could  be  placed  in  this  section. 

3.3.5  CRM  Dimensions 

After  the  crew  background  and  event  context  are  established,  the  actual  CRM 
dimensions  should  be  assessed  through  the  use  of  behavioral  markers  (as  previously 
discussed).  Behavioral  markers  should  be  rated  via  the  use  of  a  Likert  scale.  These 
behavioral  markers  should  be  referenced  back  to  CRM  curriculum  dimensions. 

3.3.6  Narrative  Summary 

Finally,  as  with  the  present  ARS  form,  a  narrative  summary  section  should  be 
included  to  facilitate  fi-ee  expression  on  the  part  of  the  reporter.  If  desired,  mnemonic 
prompts  could  be  provided  to  assist  the  reporter  in  organizing  their  thoughts  about  the  chain 
of  events.  The  NASA  ASRS  form  (Appendix  E)  provides  such  prompts  for  their 
respondents.  Pilots  and  aircrew  must  be  permitted  to  provide  open-ended  narrative 
description  of  the  event,  particularly  where  the  objective  selection  options  do  not  adequately 
capture  the  complexities  of  an  operational  CRM  situation.  The  narrative  summary  section  in 
the  current  ARS  allows  this,  but  may  necessarily  be  shortened  in  the  revised  survey  (due  to 
possible  space  limitations).  This  narrative  summary  section  should  be  analyzed  according  to 
accepted  text  analysis  methods  using  appropriate  algorithms.  These  methods  are  not 
discussed  here,  but  may  be  delineated  at  a  later  time. 
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4.  CONCLUSION  AND  RECOMMENDATIONS 


Based  on  the  findings  of  the  literature  reviewed  for  this  report,  the  USAF  and  AMC 
CRM  dimensions  concur  with  mainstream  literature  on  CRM.  The  most  defensible  USAF 
and  AMC  CRM  dimensions  include  mission,  planning,  review,  and  critique  strategies,  group 
dynamics,  workload  management,  assertiveness  training,  effective  communications,  decision 
making,  stress  awareness  and  management,  and  situational  awareness. 

For  the  rating  of  CRM  dimensions,  current  literature  supports  the  use  of  behavioral 
markers.  FAA  AC  120-5 IB  (FAA,  1995)  and  the  AMC  CRM  Assessment  Sheet  (USAF, 
1995)  provide  extensive  lists  of  CRM  behaviors.  When  appropriately  compared  and 
combined,  these  lists  provide  an  extensive  summary  of  possible  CRM  behaviors.  From  the 
combined  list,  the  individual  behavioral  markers  may  be  mapped  back  to  USAF  CRM 
dimensions.  In  this  manner,  data  gathered  from  behavioral  markers  may  be  used  to  test  the 
effectiveness  of  CRM  training  for  a  particular  core  curriculum  CRM  dimension. 

CSERIAC  recommends  the  revision  of  the  current  AMC  CRM  ARS  reporting  form. 
In  its  present  state,  the  form  does  not  appear  to  map  well  to  AFI  36-2243  (including 
Supplement  1),  accepted  taxonomies  of  CRM,  or  CRM  behavioral  markers.  Additionally, 
the  form  presents  12  factors  which  overlap  heavily  and  address  a  combination  of  CRM,  crew 
experience,  and  situational  context  issues. 

CSERIAC  recommends  that  the  ARS  form  be  extensively  revised  and  divided  into 
the  following  five  sections; 

•  Contact  information 

•  Crew  background  information 

•  Event  context  information 

•  CRM  behavioral  markers 

•  Narrative  description 

Contact  information  would  simply  include  name,  address,  and  telephone  numbers. 

The  crew  experience  section  would  include  information  on  the  crew’s  training,  proficiency, 
and  background.  The  event  context  section  would  provide  weather,  conflict,  and  equipment 
failure  information  (along  with  other  context-related  data)  of  the  situation  being  reported. 

The  CRM  behavioral  marker  section  would  address  the  CRM-specific  elements  surrounding 
the  situation  being  reported  by  allowing  the  reporter  to  rate  (on  Likert  scale)  specific, 
observable  behaviors  which  can  be  readily  traced  back  to  USAF  CRM  dimensions.  Finally, 
the  narrative  description  section  would  allow  AMC  CRM  ARS  reporters  to  freely  write  an 
account  of  the  reported  situation  from  start  to  finish,  providing  suggested  improvements 
where  appropriate. 
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APPENDIX  A 

Crew  Performance  Marker  Clusters  (from  FAA,  1995,  pp.  A1:1--A1:6) 


(Italicized  Markers  apply  to  Advanced  Technology  Flight  Decks) 

These  behavioral  markers  are  provided  to  assist  organizations  in  program  and  curriculum  development  and  to 
serve  as  guidelines  for  feedback.  They  are  not  presented  as  a  checklist  for  evaluating  individual  crewmembers. 

1.  COMMUNICATIONS  PROCESSES  AND  DECISION  BEHAVIOR  CLUSTERS 


a.  Briefings.  The  effective  briefing  is  interesting  and  thorough.  It  addresses  coordination,  planning,  and 
problems.  Although  briefings  are  primarily  a  captain’s  responsibility,  other  crewmembers  may  add  significantly 
to  planning  and  should  be  encouraged  to  do  so. 

Behavioral  Markers 

(1)  The  briefing  establishes  an  environment  for  open/interactive  communications  (for  example,  the 
captain  calls  for  questions  or  comments,  answers  questions  directly,  listens  with  patience,  does  not 
interrupt  or  “talk  over”,  does  not  rush  through  the  briefing,  and  makes  eye  contact  as  appropriate). 

(2)  The  briefing  is  interactive  and  emphasizes  the  importance  of  questions,  critique,  and  the  offering  of 
information. 

(3)  The  briefing  establishes  a  “team  concept”  (for  example,  the  captain  uses  “we”  language,  encourages 
all  to  participate  and  to  help  with  the  flight). 

(4)  The  briefing  covers  pertinent  safety  and  operational  issues. 

(5)  The  briefing  identifies  potential  problems  such  as  weather,  delays,  and  abnormal  system  operations. 

(6)  The  briefing  provides  guidelines  for  crew  actions;  division  of  labor  and  crew  workload  is  addressed. 

(7)  The  briefing  includes  the  cabin  crew  as  part  of  the  team. 

(8)  The  briefing  sets  expectations  for  handling  deviations  from  standard  operating  procedures. 

(9)  The  briefing  establishes  guidelines  for  the  operation  of  automated  systems  (for  example,  when 
systems  will  be  disabled;  which  programming  actions  must  be  verbalized  and  acknowledged). 

(10)  The  briefing  specifies  pilot  flying  and  pilot  not  flying  duties  and  responsibilities  with  regard  to 
automated  systems. 

b.  Inauirv/Advocacv/Assertion.  These  behaviors  relate  to  crewmembers’  promoting  the  course  of  action  that 
they  feel  is  best,  even  when  it  involves  conflict  with  others. 

Behavioral  Markers 

(1)  Crewmembers  speak  up  and  state  their  information  with  appropriate  persistence  until  there  is  some 
clear  resolution. 

(2)  “Challenge  and  response”  environment  is  developed. 

(3)  Questions  are  encouraged  and  are  answered  openly  and  nondefensively. 

(4)  Crewmembers  are  encouraged  to  question  the  actions  and  decisions  of  others. 

(5)  Crewmembers  seek  help  from  others  when  necessary. 

(6)  Crewmembers  question  status  and  programming  of  automated  systems  to  confirm  situational 
awareness. 

c.  Crew  Self-Critique  Regarding  Decisions  and  Actions.  These  behaviors  relate  to  the  effectiveness  of  a 
group  and/or  an  individual  crewmember  in  critique  and  debriefing.  Areas  covered  should  include  the  product, 
the  process,  and  the  people  involved.  Critique  may  occur  during  an  activity,  and/or  after  completing  it. 
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Behavioral  Markers 


(1)  Critique  occurs  at  appropriate  times,  which  may  be  times  of  low  or  high  workload. 

(2)  Critique  deals  with  positive  as  well  as  negative  aspects  of  crew  performance. 

(3)  Critique  involves  the  whole  crew  interactively. 

(4)  Critique  makes  a  positive  learning  experience.  Feedback  is  specific,  objective,  usable  and 
constructively  given. 

(5)  Critique  is  accepted  objectively  and  nondefensively. 

behaviors  related  to  free  and  open  communication.  They  reflect  the 
extent  to  which  crewmembers  provide  necessary  information  at  the  appropriate  time  (for  example  initiatina 

e^Su^oe^'^n^ developing  problems).  Active  participation  in  the  decision  making  process  is 
Ts  Snsfdered  rauMne"^  acknowledged.  Questioning  of  actions  and  decisions 

Behavioral  Markers 

(1)  Operational  decisions  are  clearly  stated  to  other  crewmembers. 

(2)  Crewmembers  acknowledge  their  understanding  of  decisions. 

(3)  Bottom  lines  for  safety  are  established  and  communicated. 

'ficluding  flight  attendants  and  others 

db  dppropnaie. 

(5)  Crewmembers  are  encouraged  to  state  their  own  ideas,  opinions,  and  recommendations. 

(6)  Efforts  are  made  to  provide  an  atmosphere  that  invites  open  and  free  communications. 

(7)  Initial  entries  and  changed  entries  to  automated  systems  are  verbalized  and  acknowledged. 

2-  TEAM  BUILDING  AND  MAINTENANCE  CLUSTER 

T[°"”ry  '°^  appropriate  leadership  and 

fotaership.  They  reflect  the  extent  to  which  the  crew  is  concerned  with  the  effective  accomplishment  of 

Behavioral  Markers 


(1)  All  available  resources  are  used  to  accomplish  the  job  at  hand. 

(2)  Fiightdeck  activities  are  coordinated  to  establish  an  acceptable  balance  between  respect  for  authority 

and  the  appropriate  practice  of  assertiveness.  ^ 

(3)  Actions  are  decisive  when  the  situation  requires. 

(4)  A  desire  to  achieve  the  most  effective  operation  possible  is  clearly  demonstrated. 

(5)  The  need  to  adhere  to  standard  operating  practices  is  recognized. 

(6)  Group  climate  appropriate  to  the  operational  situation  is  continually  monitored  and  adjusted  (for 
example,  social  conversation  may  occur  during  low  workload,  but  not  high). 

(7)  Effects  of  stress  and  fatigue  on  performance  are  recognized. 

(8)  Time  available  for  the  task  is  well  managed. 

(9)  Demands  on  resources  posed  by  operation  of  automated  systems  are  recognized  and  managed. 

(10)  When  programming  demands  could  reduce  situational  awareness  or  crew  work  overloads  levels  of 
automation  are  reduced  appropriately. 


IntsrpersoriBl  Rslationships/Group  ClimstG.  Thsse  behaviors  relate  to  the  quality  of  interpersonal 
relationships  and  the  pervasive  climate  of  the  flightdeck. 

Behavioral  Markers 

(1)  Crewmembers  remain  calm  under  stressful  conditions. 

(2)  Crewmembers  show  sensitivity  and  ability  to  adapt  to  the  personalities  of  others. 

(3)  Crewmembers  recognize  symptoms  of  psychological  stress  and  fatigue  in  self  and  in  others  (for 
example,  recognizes  when  he/she  is  experiencing  “tunnel  vision”  and  seeks  help  from  the  team;  or  notes 
when  a  crewmember  is  not  communicating  and  draws  him/her  back  into  the  team). 

(4)  “Tone”  in  the  cockpit  is  friendly,  relaxed,  and  supportive. 

(5)  During  times  of  low  communication,  crewmembers  check  in  with  others  to  see  how  they  are  doing. 

3.  WORKLOAD  MANAGEMENT  AND  SITUATIONAL  AWARENESS  CLUSTER 

a-  Preparation/Planninq/Vigilance.  These  behaviors  relate  to  crews’  anticipating  contingencies  and  the 
various  actions  that  may  be  required.  Excellent  crew  are  always  “ahead  of  the  curve”  and  generally  seem 
relaxed.  They  devote  appropriate  attention  to  required  tasks  and  respond  without  undue  delay  to  new 
developments.  (They  may  engage  in  casual  social  conversation  during  periods  of  low  workload  and  not 
necessarily  diminish  their  vigilance). 

Behavioral  Markers 

(1)  Demonstrating  and  expressing  situational  awareness;  for  example,  the  “model”  of  what  is  happening 
is  shared  within  the  crew. 

(2)  Active  monitoring  of  all  instruments  and  communications  and  sharing  relevant  information  with  the 
rest  of  the  crew. 

(3)  Monitoring  weather  and  traffic  and  sharing  relevant  information  with  the  rest  of  the  crew. 

(4)  Avoiding  “tunnel  vision”  cause  by  stress;  for  example,  stating  or  asking  for  the  “big  picture”. 

(5)  Being  aware  of  factors  such  as  stress  that  can  degrade  vigilance  and  watching  for  performance 
degradation  in  other  crewmembers. 

(6)  Staying  “ahead  of  the  curve”  in  preparing  for  planned  situations  or  contingencies. 

(7)  Ensuring  that  cockpit  and  cabin  crewmembers  are  aware  of  plans. 

(8)  Including  all  appropriate  crewmembers  in  the  planning  process. 

(9)  Allowing  enough  time  before  maneuvers  for  programming  of  the  flight  management  computer. 

(10)  Ensuring  that  all  crewmembers  are  aware  of  initial  entries  and  changed  entries  in  the  flight 
management  system. 

Workload  Distributed/Distractions  Avoided.  These  behaviors  relate  to  time  and  workload  management. 
They  reflect  how  well  the  crew  manages  to  prioritize  tasks,  share  the  workload,  and  avoid  being  distracted 
from  essential  activities. 

Behavioral  Markers 

(1)  Crewmembers  speak  up  when  they  recognize  work  overloads  in  themselves  or  in  others. 

(2)  Tasks  are  distributed  in  ways  that  maximize  efficiency. 

(3)  Workload  distribution  is  clearly  communicated  and  acknowledged. 

(4)  Non-operational  factors  such  as  social  interaction  are  not  allowed  to  interfere  with  duties. 
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(5)  Task  priorities  are  clearly  communicated. 

(6)  Secondary  operational  tasks  (for  example,  dealing  with  passenger  needs  and  communications  with 
company)  are  prioritized  so  as  to  allow  sufficient  resources  for  primary  flight  duties. 

(7)  Potential  distractions  posed  by  automated  systems  are  anticipated,  and  appropriate  preventive  action 
is  taken,  including  reducing  or  disengaging  automated  features  as  appropriate. 
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APPENDIX  B 

NASA/UT/FAA  LINE/LOS  Checklist,  Version  4 
The  Line/LOS  Checklist  Version  4  is  discussed  in: 

Helmreich,  R.  L.,  Butler,  R.  E.,  Taggart,  W.  R.,  &  Wilhelm,  J.  A.  (1995).  The  NASA/University  of 

Texas/FAA  Line/LOS  Checklist:  A  behavioral  marker-based  checklist  for  CRM  skills  assessment 
(Technical  Paper  94-02).  Austin,  TX:  NASA/University  of  Texas/FAA  Aerospace  Crew 
Research  Project. 

Additionally,  the  Line/LOS  Checklist  Version  4  has  been  provided  to  CSERIAC  courtesy  of  Mr.  John 
Wilhelm  at  the  NASA/UT/FAA  Aerospace  Crew  Research  Project.  Mr.  Wilhelm’s  contact  information 
is: 

Mr.  John  Wilhelm 

NASA/UT/FAA  Aerospace  Crew  Research  Project 

University  of  Texas  at  Austin 

1 609  Shoal  Creek  Boulevard,  Suite  200 

Austin,  Texas  78701  USA 

Telephone:  (512)480-9997 

World- wide  web  homepage: 

http://www.psy.utexas.edu/psy/helmreich/nasaut.htm 
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APPENDIX  C 

AMC  CRM  Assessment  Sheet 


Rating  (“R”)  Scale 


Poor  (1)  -  Observed  performance  is 
significantly  below  expectations.  This 
includes  situations  where  necessary 
behaviors  are  not  present  and  examples  of 
inappropriate  behavior  are  detrimental  to 
mission  effectiveness. 

Minimum  Expectations  (2) 
Observed  performance  meets  minimum 
standards,  but  there  is  room  for 
improvement.  The  level  of 

performance  is  less  than  desired  for 
effective  crew  operations. 

Standard  (3)  -  The  demonstrated 
behavior  promotes  and  maintains  crew 
effectiveness.  This  level  of 

performance  should  normally  occur  in 
day-to  day  flight  operations. 

Outstanding  (4)  -  Observed  performance 
represents  exceptional  skill  in  the 
application  of  specific  behaviors,  and 
serves  as  a  model  for  teamwork  -truly 
noteworthy  and  effective. 

Group  Dynamics 

R 

1.  Crewmembers  establish  and  maintain  a  team  concept  and  an  environment 
for  open  communications  (i.e.,  crewmembers  listen  with  patience,  do  not 
interrupt  or  "talk  over,"  do  not  rush  through  the  briefings,  &  make  eye 
contact  when  appropriate). 

2.  The  entire  crew  participates  in  briefings  as  a  team,  when  appropriate,  and 
the  crew  establishes  guidelines  for  coordination  between  all  crew  positions. 
Crewmembers  brief  and  update  passengers  when  needed  (i.e.  mx  delays, 
weather,  etc.). 

3.  Group  climate  matches  the  operational  situation  (i.e.,  presence  or  lack  of 
social  conversation).  The  crew  also  ensures  these  non-operation al  factors  do 
not  interfere  with  necessary  tasks. 

4.  The  aircraft  commander  coordinates  activities  to  establish  a  proper 
balance  between  command  authority  and  crewmember  participation.  The 
aircraft  commander  acts  decisively  when  situations  require. 

5.  Crewmembers  receive  positive  and  negative  performance  feedback  at 
appropriate  times,  and  the  atmosphere  creates  a  positive  learning  experience 
for  the  entire  crew— feedback  is  specific,  objective,  based  on  observable 
behavior,  and  given  constructively. 

6.  Crewmembers  accept  performance  feedback  objectively  and  non- 
defensively. 

7.  When  conflicts  arise,  the  crew’s  focus  remains  on  the  problem  or 
situation  at  hand.  Crewmembers  listen  actively  to  ideas  and  opinions  and 
admit  mistakes  when  wrong  (i.e.,  the  crew  resolves  conflict). 

8.  When  appropriate,  crewmembers  take  the  initiative  and  time  to  share 
operational  knowledge  and  experience  (i.e.,  new:  crewmembers,  routing, 
airfields,  situations). 

Decision  Making 

R 

1.  Crewmembers  clearly  state  operational  decisions  to  other  crewmembers 
and  receive  acknowledgment.  The  crew  includes  all  crewmembers  and 
others  when  appropriate. 

2.  Tlie  aircraft  commander  coordinates  flightdeck  activities  to  establish 
proper  balance  between  command  authority  and  crewmember  participation, 
while  acting  decisively  when  the  situation  requires. 

3.  The  crew  prepares  for  expected  and/or  contingency  situations  including 
approaches,  weather,  etc. 

4.  The  crew  remains  calm  under  stress. 

Stress  Management 

R 

1 .  Crewmembers  clearly  communicate  workload  and  task  distribution  and 
receive  acknowledgment  from  other  crewmembers.  The  crew  allots 
adequate  time  to  complete  tasks. 

2.  The  crew  prioritizes  secondary  operational  tasks  (i.e.,  dealing  with 
passenger  needs,  command  post  communications, ...)  to  retain  sufficient 
resources  to  deal  effectively  with  primary  flight  duties. 

3.  The  crew  prepares  for  expected  or  contingency  situations  including 
approaches,  weather,  etc.  (i.e.,  the  crew  stays  ahead  of  the  power  curve). 

4.  The  pilot  team  outlines  PF  and  PNF  duties  and  responsibilities  with 
regard  to  automated  systems  (i.e.,  FMS  entry  and  cross-checking). 

5.  Crewmembers  allow  sufficient  time  for  programming  of  flight 
management  computers  prior  to  maneuvers. 

6.  The  crew  uses  automated  systems  at  optimal  levels  (i.e.,  when 
programming  demands  could  reduce  situational  awareness  and  create  work 
overloads,  the  crew  reduces  the  level  of  automation  or  disengages  automated 
systems). 

7.  When  conflicts  arise,  the  crew’s  focus  remains  on  the  problem  or 
situation  at  hand.  Crewmembers  listen  actively  to  each  others’  ideas  and 
opinions  and  admit  mistakes  when  wrong  (i.e.,  the  crew  resolves  conflict). 

8.  During  long  duty  periods,  crewmembers  are  pro-active  in  remaining  alert, 
and  plan  and  use  fatigue  countermeasures. 

9.  The  crew’s  actions  do  not  create  self-imposed  stress  and  additional 
workload  (i.e.,  a  late  descent  due  to  lack  of  situational  awareness/planning). 

10.  Crewmembers  recognize  and  report  when  their  duties  or  the  duties  they 
observe  others  performing  cause  an  overload. 

1 1 .  The  crew  remains  calm  under  stress. 

Effective  Communications 

R 

1 .  The  crew’s  briefings  are  operationally  thorough,  interesting,  and  address 
crew  coordination  while  planning  for  potential  problems.  The  crew  sets 
expectations  on  how  to  handle  deviations  from  normal  operations. 

2.  Crewmembers  speak  up  and  state  their  information  with  appropriate 
persistence,  until  there  is  some  clear  resolution  and  decision  (i.e.,  effective 
advocacy  and  assertion). 

3.  Crewmembers  clearly  state  operational  decisions  to  other  crewmembers 
and  receive  acknowledgment.  Crewmembers  communicate  the  decisions  to 
the  entire  aircrew  and  others  when  appropriate. 

4.  Crewmembers  periodically  review  and  verify  the  status  of  aircraft 
automated  systems. 

5.  Crewmembers  verbalize  and  acknowledge  entries  and  changes  to 
automated  systems  parameters. 

6.  Crewmembers  allow  sufficient  time  for  programming  of  flight 
management  computers  prior  to  maneuvers. 

7.  Crewmembers  provide  positive  and  negative  performance  feedback  at 
appropriate  times  and  create  a  positive  learning  experience  for  the  whole 
crew— feedback  is  specific,  objective,  based  on  observable  behavior,  and 
constructive. 

8.  Crewmembers  accept  perfonnance  feedback  objectively  and  non- 
defensively. 

Mishap  Prevention 

R 

1 .  Crevvnmembers  check-in  with  each  other  during  times  of  high  and  low 
workload  to  maintain  situational  awareness  and  to  remain  alert. 

2.  The  crew  establishes  guidelines  for  the  operation  of  automated  systems 
(i.e.,  when  they  will  disable  systems  and  when  they  must  verbalize  and 
acknowledge  programming  actions). 

3.  The  crew  outlines  the  duties  and  responsibilities  of  the  PF  and  PNF  with 
regard  to  automated  systems  (i.e.,  FMS  entries  and  cross-checking). 

4.  Crewmembers  periodically  review  and  verify  the  status  of  aircraft 
automated  systems. 

5.  Crewmembers  verbalize  and  acknowledge  entries  and  changes  to 
automated  systems  parameters. 

6.  The  crew  plans  for  sufficient  time  for  programming  of  flight  management 
computers  prior  to  maneuvers. 

Assertiveness 

R 

1 .  Crewmembers  openly  ask  questions  regarding  crew  actions  and  decisions 
(i.e.,  effective  inquiry). 

2.  Crewmembers  speak  up  and  state  their  information  with  appropriate 
persistence,  until  there  is  some  clear  resolution  and  decision  (i.e.,  effective 
advocacy  and  assertion). 

3.  When  conflicts  arise,  the  crew's  focus  remains  on  the  problem  or  situation 
at  hand.  Crewmembers  listen  actively  to  ideas  and  opinions  and  admit 
mistakes  when  wrong  (i.e.,  the  crew  resolves  conflict). 

Overall  Observations 

R 

1.  The  crew  consistently  assesses  the  complexity  of  the  operating 
environment  (WX,  ATC,  traffic,  threat). 

2.  The  crew  assesses  the  severity  of  abnormal  systems  operation  and  other 
systems  events  during  the  mission. 

3.  Overall,  the  crew  displays  technical  proficiency. 

4.  Overall,  the  crew  effectively  performs  the  mission. 
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APPENDIX  D 
AMC  CRM  ARS  Form 

AIR  MAIL 

CREW  RESOURCE  MANAGEMENT 
ANONYMOUS  REPORTING  SYSTEM 

Air  Mail  provides  you  with  a  way  to  improve  Crew  Resource  Management  training.  Fill  out  the  form  when 
you  feel  you  have  experienced  or  observed  a  positive  or  negative  example  of  CRM.  Your  inputs  will  be 
sanitized  and  sent  to  our  CRM  training  contractors  and  “blue-suit”  trainers  for  use  in  CRM  programs.  This 
program  is  strictly  anonymous  unless  you  want  to  provide  more  information  or  wish  to  be  contacted  about 
this  incident.  If  so,  include  your  name  and  DSN  phone  number.  HQ  AMC  guarantees  crewmembers 
complete  anonymity  and  non-retribution.  This  guarantee  applies  to  all  data  from  AIR  MAIL  reports.  This 
guarantee  does  not  apply  to  similar  information  gleaned  from  other  sources  or  information  relating  to 
criminal  conduct.  This  form  does  not  require  postage,  even  if  you  attach  additional  sheets  to  the  original. 
Sharing  your  experience  will  enhance  flight  safety. 

fold  here  last 


NO  POSTAGE 
NECESSARY 
IF  MAILED 
IN  THE 

UNITED  STATES 


BUSINESS  REPLY  MAIL 

FIRST-CLASS  MAIL  PERMIT  NO.  867  BELLEVILLE  IL 


POSTAGE  WILL  BE  PAID  BY  ADDRESSEE 

AIR  MAIL 

PO  BOX  25285 

SCOTT  AFB  IL  62225-9904 


fold  here  first 


PLEASE  TAKE  TIME  TO  COMPLETE  THE  FOLLOWING: 

Your  answers  to  the  following  will  be  entered  into  a  CRM  trend  analysis  data  base  and  used  to  improve 
operations  and  training  programs.  Please  check  any  factors  you  feel  contributed  to  the  incident. 


□  1 .  Interruption  in  a  planned  activity 

□ 

7.  Inadequate  planning 

□  2.  Non-standard/misunderstood  communications 

□ 

8.  Crew  coordination 

□  3.  Proficiency 

□ 

9.  Equipment  failure 

□  4.  Misleading/erroneous  guidance  or  manuals 

□ 

10.  Visual  illusion/spatial  disorientation 

□  5.  High  workload 

□ 

1 1 .  Insufficient  training 

□  6.  Complacency 

□ 

12.  Stress  or  anxiety 

Other; _ 

AMC  FORM  38,  OCT  93 
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AIR  MAIL  IS  A  TOTALLY  ANONYMOUS  INFORMATION  GATHERING  PROGRAM. 

Provide  as  much  information  as  possible  about  what  happened.  Specific  names,  locations  and  dates  are 
not  important.  Facts  such  as  aircraft  type,  crew  positions,  and  phase  of  flight  are  important  to  clearly 
understand  the  circumstances.  Information  that  could  be  used  to  pinpoint  an  office  or  individual  will  be 
removed  or  changed  before  release.  Tell  us  about  both  poor  and  exceptional  CRM  performance.  If 
more  space  is  needed,  use  a  blank  sheet  of  paper  and  attach  inside  this  form  before  mailing. 

Aircraft  type _  Crew  Position _  Phase  of  flight _ 


APPENDIX  E 
NASA  ASRS  Form 


E-1 


DO  NOT  REPORT  AIRCRAFT  ACCIDENTS  AND  CRIMINAL  ACTIVITIES  ON  THIS  FORM, 

ACCIDENTS  AND  CRIMINAL  ACTIVITIES  ARE  NOT  INCLUDED  IN  THE  ASRS  PROGRAM  AND  SHOULD  NOT  BE  SUBMITTED  TO  NASA 
ALL  IDENTITIES  CONTAINED  IN  THIS  REPORT  WILL  BE  REMOVED  TO  ASSURE  COMPLETE  REPORTER  ANONYMITY. 

(SPACE  BELOW  RESERVED  FOR  ASRS  DATE/TIME  STAMP) 

IDENTIFICATION  STRIP:  P/ease  fill  in  all  blanks  to  ensure  return  of  strip. 

NO  RECORD  WILL  BE  KEPT  OF  YOUR  IDENTITYJhis  section  will  be  returned  to  you. 

TELEPHONE  NUMBERS  where  we  may  reach  you  for  further 


details  of  this  occurrence: 

HOME  Area  No. 

_  Hours 

WORK  Area  No. 

_  Hours 

NA  ME _  type  of  EVENT/SITUATION 

ADDRESS/PO  BOX _  _ 

_  DATE  OF  OCCURRENCE _ 

CITY _ STATE _ ZIP _  LOCAL  TIME  (24  hr.  clock)  _ 


PLEASE  FILL  IN  APPROPRIATE  SPACES  AND  CHECK  ALL  ITEMS  WHICH  APPLY  TO  THIS  EVENT  OR  SITUATION. 


REPORTER 

FLYING  TIME 

CERTIFICATES/RATINGS 

ATC  EXPERIENCE  1 

0  Captain 

total 

hrs. 

0  student 

o  private 
oATP 

oFPL 

0  Developmental 

vrs. 

0  First  Officer 
o  pilot  flying 

0  pilot  not  flying 

0  Other  Crewmember 

0  commercial 

radar 

last  90  days 

_hrs. 

0  instrument 

oCFI 

non-radar 

vrs. 

0  multiengine 

oF/E 

supervisory 

vrs. 

o 

time  in  type 

.  hrs. 

0 

military  _ 

vrs. 

AIRSPACE 

WEATHER 

LIGHTA/ISIBILITY 

ATC/ADVISORYSERV. 

o  Class  A  (PCA) 

0  Class  B  (TCA) 
o  Class  C  (ARSA) 

0  Class  D  (Control  Zone/ATA) 

0  Class  E  (General  Controlled) 
o  Class  G  (Uncontrolled) 

o  Special  Use  Airspace 

0  airwav/route 
o  unknown/other 

o  VMC  o  ice 

0  IMC  o  snow 

o  mixed  o  turbulence 

o  marginal  o  tstorm 

orain  owindshear 

ofog  o 

0  daylight 

0  dawn 

ceiling  _ 

visibility  _ 

RVR 

o  night 

0  dusk 

feet 

_ miles 

feet 

o  local  0  center 

©ground  o  FSS 

0  apch  o  UNICOM 

0  dep  0  CTAF 

Name  of  ATC  Facility: 

AIRCRAFT  1 

AIRCRAFT  2  I 

Type  of  Aircraft 
(Make/Model) 

(Your  Aircraft)  . 

©EFIS 
©  FMS/FMC 

(Other  Aircraft) 

©EFIS 
©  FMS/FMC 

Operator 

0  air  carrier 

0  commuter 

©  military 
©  private 

©  corporate 
©  other 

©  air  carrier 
©  commuter 

©  military 
©  private 

©  corporate 
©  other 

Mission 

o  passenger 
o  cargo 

©  training 
©  pleasure 

©  business 
©  unk/other 

©  passenger 
©cargo 

©  training 
©  pleasure 

©  business 
©  unk/other 

Flight  plan 

oVFR 

olFR 

©SVFR 

©DVFR 

©  none 
©  unknown 

©VFR 

©IFR 

©SVFR 

©DVFR 

©  none 
©  unknown 

Flight  phases  at 
time  of  occurrence 

©taxi 
©  takeoff 
©  climb 

©  cruise 
©  descent 
©  approach 

©  landing 

©  missed  apch/GAR 
©  other 

©taxi 
©  takeoff 

0  climb 

©  cruise 
©  descent 
©  approach 

©  landing 

©  missed  apch/GAR 
0  other 

Control  status 

©  visual  apch 
©  controlled 
©  no  radio 

©  on  vector  ©  on  SID/STAR 

©  none  ©  unknown 

©  radar  advisories 

©  visual  apch 
©  controlled 
©  no  radio 

©  on  vector  ©  on  SI  D/STAR 

©  none  o  unknown 

©  radar  advisories 

If  more  than  two  aircraft  were  involved,  please  describe  the  additional  aircraft  in  the  “Describe  Event/Situation”  section. 


LOCATION 

CONFLICTS 

Altitude  0  MSL  oAGL 

Estimated  miss  distance  in  feet:  horiz 

vert 

Distance  and  radial  from  airport,  NAVAID,  or  other  fix 

Was  evasive  action  taken? 

©Yes 

©  No 

Was  TCAS  a  factor?  ©  TA 

©RA 

©  No 

Nearest  CItv/State 

Did  GPWS  activate? 

©Yes 

©  No 

NASA  ARC  277B  (January  1994) 
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NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 

NASA  has  established  an  Aviation  Safety  Reporting  System  (ASRS)  to 
identify  issues  in  the  aviation  system  which  need  to  be  addressed.  The 
program  of  which  this  system  Is  a  part  is  described  in  detail  in  FAA 
Advisory  Circular  00-46C.  Your  assistance  in  informing  us  about  such 
issues  Is  essential  to  the  success  of  the  program.  Please  fill  out  this  form 
as  completely  as  possible,  enclose  In  an  sealed  envelope,  affix  proper 
postage,  and  and  send  it  directly  to  us. 

The  information  you  provide  on  the  identity  strip  will  be  used  only  if  NASA 
determines  that  it  is  necessary  to  contact  you  for  further  Information. 
THIS  IDENTITY  STRIP  WILL  BE  RETURNED  DIRECTLY  TO  YOU.  The 
return  of  the  identity  strip  assures  your  anonymity. 

NOTE: 


AVIATION  SAFETY  REPORTING  SYSTEM 

Section  91.25  of  the  Federal  Aviation  Regulations  (14  CFR  91.25) 
prohibits  reports  filed  with  NASA  from  being  used  for  FAA  enforcement 
purposes.  This  report  will  not  be  made  available  to  the  FAA  for  civil 
penalty  or  certificate  actions  for  violations  of  the  Federal  Air  Regulations. 
Your  identity  strip,  stamped  by  NASA,  is  proof  that  you  have  submitted  a 
report  to  the  Aviation  Safety  Reporting  System.  We  can  only  return  the 
strip  to  you,  however,  if  you  have  provided  a  mailing  address.  Equally 
important,  we  can  often  obtain  additional  useful  information  if  our  safety 
analysts  can  talk  with  you  directly  by  telephone.  For  this  reason,  we  have 
requested  telephone  numbers  where  we  may  reach  you. 

Thank  you  for  your  contribution  to  aviation  safety. 


AIRCRAFT  ACCIDENTS  SHOULD  NOT  BE  REPORTED  ON  THIS  FORM.  SUCH  EVENTS  SHOULD  BE  FILED  WITH  THE 
NATIONAL  TRANSPORTATION  SAFETY  BOARD  AS  REQUIRED  BY  NTSB  Regulation  830.5  (49CFR830.5). 


Please  fold  both  pages  (and  additional  pages  if  required),  enclose  in  a  sealed,  stamped  envelope,  and  mail  to: 

^  A  NASA  AVIATION  SAFETY  REPORTING  SYSTEM 

POST  OFFICE  BOX  1 89 

r  N  MOFFETT  FIELD,  CALIFORNIA  94035-0189 


DESCRIBE  EVENT/SITUATION 


keeping  in  mind  the  topics  shown  below,  discuss  those  which  you  feel  are  relevant  and  anything  else  you  think  is  important.  Include  what  you  believe  really  caused  the 
brobiem,  and  what  can  be  done  to  prevent  a  recurrence,  or  correct  the  situation.  (  USE  ADDITIONAL  PAPER  IF  NEEDED) 


CHAIN  OF  EVENTS 

-  How  the  problem  arose  >  How  it  was  discovered 

-  Contributing  factors  -  Corrective  actions 
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HUMAN  PERFORMANCE  CONSIDERATIONS 

-  Perceptions,  judgments,  decisions  -  Actions  or  inactions 

-  Factors  affecting  the  quality  of  human  performance 


/  • 


[NOTE:  NOTIOhlAll  TO  INITIATE  THE  ITERATIVE  TAILORING  PROCESS  ] 

AMC  CREW  RESOURCE  MANAGEMENT 
ANONYMOUS  REPORTING  SYSTEM  (AMC  CRM  ARS) 


FIRST  FOLD;  FOLD  FORM  IN  HALF,  FOLDING  BOTTOM  TO  TOP  OF  THIS  SIDE  . 


The  AMC  CRM  ARS  provides  you  with  a  way  to  improve  Crew  Resource  Management  (CRM) 
training.  Fill  out  the  form  when  you  feel  you  have  experienced  or  observed  a  positive  or  negative 
example  of  CRM,  Your  inputs  will  be  sanitized  (i.e.,  information  leading  to  your  association  with 
the  reported  event  will  be  removed)  and  sent  to  our  CRM  training  contractors  and  “blue-suit” 
trainers  for  use  in  CRM  programs.  This  program  is  strictly  anonymous  unless  you  want  to  provide  more  information  or  wish  to  be  contacted  about 
this  event.  If  so,  please  complete  the  eontact  information  below.  Contact  data  will  not  be  archived  and  the  contact  information  you  provide  will  be 
returned  to  you.  HQ  AMC  guarantees  crewmembers  complete  anonymity  and  non-retribution.  This  guarantee  applies  to  all  data  from  AMC  CRM 
ARS  reports.  This  guarantee  does  not  apply  to  similar  information  gleaned  from  other  sources  or  information  relating  to  criminal  conduct.  This 
form  does  not  require  postage,  even  if  you  attach  additional  sheets  to  the  original.  Sharing  your  experience  will  enhance  flight  safety. 

(OPTIONAL) 

Your  Name 


HomeAVork  Phone: 


Address: 


City/State/ZIP:_ 


Type  (e.g.,  a  “Title”)  of  Event:_ 
Date/Local  Time  of  Event: _ 


Crew  Background:  Please  respond  with  the  most  accurate  information  you  have  available  (print  N/A  where  data  is  not  applicable,  and  UN  if  unknown). 


Crew  Member  Total  Hours  Hours  In  Type 

Hours/90  Day  Last  CRM  Train  Date 

Special  CRM  Training  Type 

Instructor  Pilot 

Type  IG  Type  2G  Type  3G 

Pilot 

Type  IG  Type  2G  Type  3G 

Copilot 

Type  IG  Type  2G  Type  3G 

Navigator 

Type  IG  Type  2G  Type  3G 

Aft  Crew 

Type  IG  Type  2G  Type  3G 

ATC  Specialist  ^  ^ | 

j 

Type  IG  Type  2G  Type  3G 

CREWMEMBER  FLYING  WAS 

(Pilot,  Copilot) 

Event  Context:  Please  respond  to  ^  that  apply. 


Your  Aircraft/Flight  Phase/Location:  Type _ Mission _ Flight  Phase:  TaxiQ  TakeofQ 

ClimbG  CruiseQ  DescenO  ApproachQ  LandingG  Missed  ApchQ  OtherQ _ Altitude _ MSLG  AGLQ 

Nearest  NAV  Fix _  Distance  from  Fix _ (Miles),  Radial _ °FROM 

Weather/LightA^isibility:  IMCQ  VMCQ  MixedQ  MarginaO  IceQ  SnowQ  RainQ  FogQ  TStormQ  TurbulenceQ  WindshearQ 

DaylighO  DuskQ  NightQ  DawnQ  Ceiling _ (Feet),  Visibility _ (Miles).  RVR _ (Feet). 

Airspace/Air  Traffic  Control/Advisory  Services:  MOAQ  IR/VR  Training  RouteQ# _ ADIZQ  Airspace  Class:  AQ  BQ 

CQ  DO  £□  GG  ATC:  Cine  DeO  GroundG  TowerG  DepG  CenterG  ApchG  FSSG  UNICOMG  CTAFG  ATC  Facility 

Name _  Control  Status:  Visual  ApchG  ControlledG  NoRadG  NoneG  On  VectorG  Radar  AdvisoriesG  SID/STARG 

Conflicts  (Please  describe  situation  completely  in  narrative  summary):  Miss  distance  (feet):  Horiz _ Vert _ 


CRM  Behaviors:  Please  rate  the  following  behaviors  on  a  scale  of  1  (poor)  to  4  (outstanding)  as  described  below. 


Description  of  4-Point  Likert  Rating  Scale 

1  (Poor):  Observed  performance  is  significantly  below  expectations. 

2  (Minimum  Expectations):  Observed  performance  meets  minimum 
standards,  but  there  is  room  for  improvement. 

3  (Standard):  The  documented  behavior  promotes  and  maintains  crew 
effectiveness. 

4  (Outstanding):  Observed  performance  represents  exceptional  skill  in  the 
application  of  specific  behaviors,  and  serves  as  a  model  for  teamwork — truly 
noteworthy  and  effective. 


MPR&CS:  Mission,  Planning,  Review  and  Critique  Strategies  Rating 


1. 

The  crew’s  briefings  were  operationally  thorough,  interesting,  and 
addressed  crew  coordination  while  planning  for  potential  problems. 

1. 

2. 

The  briefing  established  an  environment  for  open/interactive 
communications. 

2. 

3. 

The  briefing  was  interactive  and  emphasized  the  importance  of 
questions,  critique,  and  the  offering  of  information. 

3. 

4. 

The  briefing  established  a  “team  concept." 

4. 

5. 

The  briefing  covered  pertinent  safety  and  operational  issues. 

5. 

6. 

The  briefing  identified  potential  problems  (weather,  delays,  and 
abnormal  system  operations). 

S. 

7. 

The  briefing  provided  guidelines  for  crew  actions  (division  of  labor 
and  crew  workload). 

7. 

8. 

The  briefing  included  the  cabin  crew  as  part  of  the  team. 

8. 

9. 

The  briefing  set  expectations  for  handling  deviations  from  standard 
operating  procedures. 

9. 

10. 

The  briefing  established  guidelines  for  the  operation  of  automated 
systems. 

10. 

11. 

The  briefing  specified  pilot  flying  and  pilot  not  flying  duties  and 
responsibilities  with  regard  to  automated  systems. 

11. 

12. 

When  appropriate,  crew  members  shared  operational  knowledge 
and  experience. 

12. 

13. 

Critique  occurred  at  appropriate  times  (which  could  be  times  of  low 
or  high  workload). 

13. 

14. 

Critique  dealt  with  positive  and  negative  aspects  of  crew 
performance 

14. 

15. 

Critique  involved  the  whole  crew  interactively. 

15. 

16. 

Critique  made  a  positive  learning  experience.  Feedback  was 
specific,  objective,  usable  and  constructive. 

16. 

17. 

Critique  was  accepted  objectively  and  nondefensively. 

17. 

WM 

Workload  Management 

Rating 

1. 

The  need  to  adhere  to  standard  operating  practices  was 
recognized. 

2. 

Time  available  for  the  task  was  well  managed. 

2. 

3. 

Resource  demands  posed  by  operation  of  automated  systems 
were  recognized  and  managed. 

3. 

4. 

The  crew  used  automated  systems  at  optima!  levels. 

4. 

5. 

The  crew  prepared  for  expected  or  contingency  situations  including 
approaches,  weather,  etc. 

5. 

6. 

The  crew  allowed  sufficient  time  for  programming  flight 
management  computers  prior  to  maneuvers. 

5. 

7. 

The  crew’s  actions  did  not  create  self-imposed  stress  and 
additional  workload. 

7. 

8. 

Crew  members  recognized  and  reported  when  their  duties  or  the 
duties  they  observed  others  performing  caused  an  overload. 

B. 

9. 

Tasks  were  distributed  in  ways  that  maximized  efficiency. 

9. 

10. 

Crew  members  communicated  workload  and  task  distribution  and 
received  acknowledgment  from  other  crew  members. 

10. 

11. 

Task  priorities  were  clearly  communicated. 

11. 

12. 

The  crew  prioritized  secondary  operational  tasks  (i.e.,  dealing  with 
passenger  needs,  command  post  communications)  to  retain 
sufficient  resources  to  deal  with  primary  task  duties. 

12. 

13. 

Potential  distractions  posed  by  automated  systems  were 
anticipated,  and  appropriate  preventive  actions  were  taken,  (i.e., 
reducing  or  disengaging  automated  features  as  appropriate). 

13. 

14. 

Crew  members  checked  in  with  each  other  during  times  of  high/low 
workload  to  maintain  situational  awareness  and  to  remain  alert. 

14. 

EC: 

Effective  Communications 

Rating 

1. 

The  “big  picture”  was  shared  within  the  team,  including  others  as 
appropriate. 

1. 

~2. 

Crewmembers  were  encouraged  to  state  their  own  ideas,  opinions, 
and  recommendations. 

2. 

“3^ 

Efforts  were  made  to  provide  an  atmosphere  that  invited  open  and 
free  communications. 

3. 

~ 

Initial  entries  and  changed  entries  to  automated  systems  were 
verbalized  and  acknowledged. 

4. 

”5^ 

During  times  of  low  communication,  crewmembers  checked  in  with 
others. 

5. 

_ 

Cockpit  and  cabin  crewmembers  were  aware  of  plans. 

S. 

7. 

Crew  members  verbalized/acknowledged  entries/changes  to 
automated  systems  parameters. 

7, 

f )  * 


[NOTE:  NOTIONAL  'f6rM  TO  INITIATE  THE  ITERATIVE  TAILORING  PROCESS 

Second  Fold:  Fold  form  rN  half  again,  ensuring  that  this  side  is  visible, 


No  Postage 
Necessary  if 
Mailed  IN  THE 
United  States 


BUSINESS  REPLY  MAIL 

First-class  Mail  Permit  No.  867  Belleville,  IL 


Postage  will  be  paid  by  addressee. 

AIR  MAIL 

PO  BOX  25285 

SCOTT  AFB,  IL  62225-9904 


Please  continue  rating  these  behaviors  on  a  scale  of  I  (poor)  to  4  (outstanding). 


GD:  Group  Dynamics _  Rating 


1. 

Crewmembers  sought  help  from  others  when  necessary. 

2. 

When  conflicts  arose,  the  crew’s  focus  remained  on  the  problem  or 
situation  at  hand.  Crew  members  listened  actively  to  ideas  and 
opinions  and  admitted  mistakes  when  wrong. 

2. 

3. 

“Tone"  in  the  cockpit  was  friendly,  relaxed,  and  supportive. 

3. 

4. 

All  available  resources  were  used  to  accomplish  the  job  at  hand. 

4. 

5. 

The  aircraft  commander  coordinated  flightdeck  activities  to 
establish  proper  balance  between  command  authority  and  crew 
member  participation. 

5. 

6. 

The  aircraft  commander  acted  decisively  when  the  situation 
required. 

6, 

7. 

A  desire  to  achieve  the  most  effective  operation  possible  was 
clearly  demonstrated. 

7. 

8. 

Group  climate  appropriate  to  the  operational  situation  was 
continually  monitored  and  adjusted. 

8. 

9. 

Crewmembers  showed  sensitivity  and  the  ability  to  adapt  to  the 
personalities  of  others. 

9. 

10. 

All  appropriate  crewmembers  were  included  in  the  planning 
process. 

10. 

11. 

Group  climate  matched  the  operational  situation  (i.e.,  presence  or 
lack  of  social  conversation).  The  crew  also  ensured  these  non- 
operational  factors  did  not  interfere  with  necessary  tasks. 

11. 

SA:  Situational  Awareness _  Rating 


1 .  The  crew  demonstrated  and  expressed  situational  awareness.  For 
example,  the  “model”  of  what  is  happening  was  shared  within  the 
crew. 

1. 

2.  The  crew  actively  monitored  all  instruments  and  communications 
and  shared  relevant  information  with  the  rest  of  the  crew. 

2. 

3.  The  crew  consistently  assessed  the  complexity  of  the  operating 
environment  {Weather,  ATC,  traffic,  threats). 

3. 

DM: 

Decision  Making 

Rating 

1. 

“Bottom  lines"  for  safety  were  established  and  communicated. 

1. 

2. 

The  crew  assessed  the  severity  of  abnormal  systems  operation 
and  other  systems  events  during  the  mission. 

2. 

3. 

Operational  decisions  were  clearly  stated  to  other  crewmembers. 

3. 

H 

Crewmembers  acknowledged  their  understanding  of  decisions. 

4. 

AS: 

Assertiveness  Training 

Rating 

1. 

Crewmembers  spoke  up  and  stated  their  information  with 
appropriate  persistence  until  there  was  some  clear  resolution. 

2. 

“Challenge  and  response”  environment  was  developed. 

2. 

3. 

Questions  were  encouraged  and  answered  openly  and 
nondefensively. 

3. 

4. 

Crewmembers  were  encouraged  to  question  the  actions  and 
decisions  of  others. 

4. 

5. 

Crewmembers  questioned  the  status  and  programming  of 
automated  systems  to  confirm  SA. 

5. 

SM;  Stress  Awareness  and  Management 

Rating 

1. 

Effects  of  stress  and  fatigue  on  performance  were  recognized. 

2. 

Crewmembers  remained  calm  under  stressful  conditions. 

2. 

3. 

Crewmembers  recognized  symptoms  of  psychological  stress  and 
fatigue  in  self  and  in  others. 

3. 

4. 

Crewmembers  avoided  “tunnel  vision"  caused  by  stress:  for 
example,  crewmembers  stated  or  asked  for  the  “big  picture". 

4. 

5. 

During  long  duty  periods,  crew  members  were  pro-active  in 
remaining  alert,  and  planned  and  used  fatigue  countermeasures. 

5. 

6. 

Crewmembers  were  aware  of  factors  that  could  degrade  vigilance 
and  watched  for  performance  degradation  in  other  crewmembers. 

B, 

NOTE:  SOME  SUBSET  OF  THESE  ITEMS  SHOULD  BE  CHOSEN 
BASED  ON  DESIRED  COMMAND  EMPHASIS _ 


Narrative  Summary:  Please  describe  the  CRM  event  you  are  reporting.  Attach  additional  sheets  to  side  I  (inside  of  the  folded  form). 


APPENDIX  F 

Vision  of  Possible  ARS  Form 


F-1 


About  CSERIAC 


The  Crew  System  Ergonomics  Information  Analysis  Center  (CSERIAC)  is  the  gate¬ 
way  to  worldwide  sources  of  up-to-date  human  factors  and  ergonomics  information 
and  technologies  for  designers,  engineers,  researchers,  and  human  factors  special¬ 
ists.  CSERIAC  provides  a  variety  of  products  and  services  to  government,  industry, 
and  academia  promoting  the  use  of  human  factors  and  ergonomics  in  the  design  of 
human-operated  equipment  and  systems. 


CSERIAC’s  primary  objective  is  to  acquire,  analyze,  and  disseminate  timely  informa¬ 
tion  on  human  factors  and  ergonomics.  On  a  cost-recovery  basis,  CSERIAC  will: 

■  Distribute  human  factors  and  ergonomics  technologies  and  publications 

■  Perform  customized  bibliographic  searches  and  reviews 

■  Prepare  state-of-the-art  reports  and  critical  reviews 

■  Conduct  specialized  analyses  and  evaluations 

■  Organize  and  conduct  workshops  and  conferences 


CSERIAC  is  a  Department  of  Defense  Information  Analysis  Center  sponsored  by  the 
Defense  Technical  Information  Center.  It  is  technically  managed  by  the  Armstrong 
Laboratory  Human  Engineering  Division  and  operated  by  the  University  of  Dayton 
Research  Institute. 
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